Wearable apparatus for analyzing group dynamics

ABSTRACT

A wearable apparatus and methods may analyze images. In one implementation, a wearable apparatus for capturing and processing images may comprise a wearable image sensor configured to capture a plurality of images from an environment of a user of the wearable apparatus and at least one programming device. The at least one processing device may be programmed to: perform a first analysis of the plurality of images to detect at least two persons; perform a second analysis of the plurality of images to determine association information related to the at least two detected persons; and update a social representation based on the determined association information.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/488,789, filed Apr. 23, 2017; U.S. ProvisionalPatent Application No. 62/488,791, filed Apr. 23, 2017; and U.S.Provisional Patent Application No. 62/503,034, filed May 8, 2017. All ofthe foregoing applications are incorporated herein by reference in theirentirety.

BACKGROUND Technical Field

This disclosure generally relates to devices and methods for capturingand processing images from an environment of a user, and usinginformation derived from captured images.

Background Information

Today, technological advancements make it possible for wearable devicesto automatically capture images and store information that is associatedwith the captured images. Certain devices have been used to digitallyrecord aspects and personal experiences of one's life in an exercisetypically called “lifelogging.” Some individuals log their life so theycan retrieve moments from past activities, for example, social events,trips, etc. Lifelogging may also have significant benefits in otherfields (e.g., business, fitness and healthcare, and social research).Lifelogging devices, while useful for tracking daily activities, may beimproved with capability to enhance one's interaction in his environmentwith feedback and other advanced functionality based on the analysis ofcaptured image data.

Even though users can capture images with their smartphones and somesmartphone applications can process the captured images, smartphones maynot be the best platform for serving as lifelogging apparatuses in viewof their size and design. Lifelogging apparatuses should be small andlight, so they can be easily worn and would not require intensiveattention and operation from the user. Moreover, with improvements inimage capture devices, including wearable apparatuses, additionalfunctionality may be provided to assist users in navigating in andaround an environment, identifying persons and objects they encounter,and providing feedback to the users about their surroundings andactivities. Therefore, there is a need for apparatuses and methods forautomatically capturing and processing images to provide usefulinformation to users of the apparatuses, and for systems and methods toprocess and leverage information gathered by the apparatuses.

SUMMARY

Embodiments consistent with the present disclosure provide devices andmethods for automatically capturing and processing images from anenvironment of a user, and systems and methods for processinginformation related to images captured from the environment of the user.

In one embodiment, a wearable apparatus for capturing and processingimages may comprise a wearable image sensor configured to capture aplurality of images from an environment of a user of the wearableapparatus; and at least one processing device. The at least oneprocessing device may be programmed to: analyze the plurality of imagesto identify a plurality of people; analyze the plurality of images todetermine an affinity level between the user and each of the pluralityof people; obtain an image representation of each of the plurality ofpeople; and generate, based on the affinity levels, a visualizationcomprising the image representations.

In one embodiment, a wearable apparatus may comprise a wearable imagesensor configured to capture a plurality of images from an environmentof a user of the wearable apparatus; and at least one processing device.The at least one processing device may be programmed to: receive fromthe wearable image sensor a plurality of facial images of individualswith whom the user interacted; store, in association with each of theplurality of facial images, at least one linking attribute, the at leastone linking attribute including at least one of an environmentalcharacteristic, a geographical location, an associated word, a socialinterconnection, or a temporal indicator; and cause a visualrepresentation to be displayed. The visual representation may includethe plurality of facial images, and the plurality of facial images maybe arranged based on the at least one linking attribute.

In one embodiment, a non-transitory computer-readable medium for use ina system employing a wearable image sensor pairable with a mobilecommunications device, the computer readable medium containinginstructions. The instructions, when executed by at least one processor,may cause the at least one processor to perform steps, comprising:receiving, from the wearable image sensor worn by a user, a plurality offacial images of individuals with whom the user interacted; storing, ina manner associated with each of the plurality of facial images, atleast one linking attribute, the at least one linking attributeincluding at least one of an environmental characteristic, ageographical location, an associated word, a social interconnection, ora temporal indicator; and displaying, in a face cloud, the plurality offacial images. The plurality of facial images may be arranged based onthe at least one linking attribute.

In one embodiment, a non-transitory computer-readable medium containsinstructions. The instructions, when executed by at least one processormay cause the at least one processor to perform steps, comprising:receiving from a wearable image sensor worn by a user a plurality offacial images of individuals with whom the user has interacted;receiving a list of words parsed from a sound file captured during atime period associated with detection of a presence of the plurality ofindividuals in a vicinity of the wearer; associating in memory, the listof parsed words with a corresponding individual; and presenting on ascreen of a display a subset of the plurality of facial images ofindividuals whose word lists contain overlapping common words.

In one embodiment, a non-transitory computer-readable medium may containinstructions that, when executed by at least one processor cause the atleast one processor to perform steps. The steps may comprise receivingfacial images of a plurality of individuals with whom a user interacted;receiving at least one linking attribute for each of the plurality offacial images; associating in memory each of the plurality of facialimages with the at least one linking attribute received for acorresponding one of the plurality of facial images; and for a selectedlinking attribute, presenting on a screen of a display a set of theplurality of facial images that share the selected linking attribute.

In one embodiment, a non-transitory computer-readable medium may containinstructions that, when executed by at least one processor cause the atleast one processor to perform steps. The steps may comprise receivingfrom a wearable image sensor worn by a user at least one image capturedin a time window; receiving a file of sound captured in a vicinity ofthe image sensor during the time window; processing the sound file toidentify at least one word in the sound file; storing in memory anassociation between the word and the object image; receiving at a timeremote from the time window a search input of the at least one key word;accessing the memory to locate the association between the at least onekey word and the at least one object image; and causing a display of theobject image on a mobile communications device of the user, separatefrom the wearable image sensor.

In one embodiment, a non-transitory computer-readable medium may containinstructions that, when executed by at least one processor cause the atleast one processor to perform steps. The steps may comprise receivingfrom a wearable image sensor worn by a user a first plurality of facialimages of individuals with whom the user interacted on a first calendardate; storing an indicator of the first calendar date in associationwith each of the first plurality of individuals; receiving from thewearable image sensor worn by the user a second plurality of facialimages of individuals with whom the user interacted on a second calendardate; storing an indicator of the second calendar date in associationwith each of the second plurality of individuals; receiving a firstinput request by the user for recall of interactions on the firstcalendar date; in response to the first input, displaying to the userimages of at least some of the first plurality of individuals; receivinga second input request by the user for recall of interactions on thesecond calendar date; and in response to the second input, displaying tothe user images of at least some of the second plurality of individuals.

In one embodiment, a wearable apparatus for capturing and processingimages may comprise a wearable image sensor configured to capture aplurality of images from an environment of a user of the wearableapparatus and at least one processing device. The at least oneprocessing device may be programmed to analyze the plurality of imagesto detect at least one person; select at least one record associatedwith the detected person; analyze the plurality of images to identifycontextual information associated with the detected person; and updatethe at least one record based on the contextual information.

In one embodiment, a wearable apparatus for capturing and processingimages may comprise a wearable image sensor configured to capture aplurality of images from an environment of a user of the wearableapparatus and at least one processing device. The at least oneprocessing device may be programmed to analyze the plurality of imagesto detect at least one person; determine contextual category based onthe detected person; analyze the plurality of images to identify avisual context indicator; and associate the visual context indicatorwith the determined contextual category.

In one embodiment, a wearable apparatus may comprise a wearable imagesensor configured to capture a plurality of images from an environmentof a user and at least one processor. The at least one processor may beprogrammed to determine from at least one of the plurality of images apresence of two differing objects indicated by the user; perform a lookup of the two differing objects to ascertain identities of the twodiffering objects; perform a look up of descriptive information aboutthe two differing objects; compare the descriptive information about thetwo differing objects; and cause a display of the information about thecomparing of the two differing objects in a manner permitting the userto ascertain differences between the two differing objects.

In one embodiment, a method, may comprise receiving, by a processor, aplurality of images captured by a wearable image sensor from anenvironment of a user; and determining from at least one of theplurality of images a presence of two differing objects indicated by theuser; performing a look up of the two differing objects to ascertainidentities of the two differing objects; performing a look up ofdescriptive information about the two differing objects; comparing thedescriptive information about the two differing objects; and causing adisplay of the information about the comparing of the two differingobjects in a manner permitting the user to ascertain differences betweenthe two differing objects.

In one embodiment, a wearable apparatus may comprise a wearable imagesensor and at least one processor. The at least one processor may beprogrammed to: receive, from the wearable image sensor, a facial imageof an individual with whom a user of the wearable apparatus interactedin a first interaction during a time window; receive sound data capturedin a vicinity of the image sensor during at least a part of the timewindow; process the sound data to identify at least one key word; storein memory an association between the key word and the facial image;receive, from the wearable image sensor, another facial image of theindividual during a second interaction at a time other than during thetime window; use image processing to determine that the individual inthe first interaction is the individual in the second interaction;access the memory to locate the at least one key word from the firstinteraction; and during the second interaction, cause a display of atleast one key word on a display visible to the user, to thereby remindthe user of subject matter of the first interaction.

In one embodiment, a non-transitory computer-readable medium may be foruse in a system employing a wearable image sensor pairable with a mobilecommunications device. The computer-readable medium may containinstructions that when executed by at least one processor cause the atleast one processor to perform steps. The steps may comprise receivingfrom the wearable image sensor worn by a user a facial image of anindividual with whom the user interacted in a first interaction during atime window; receiving sound data captured in a vicinity of the imagesensor during the time window; processing the sound data to identify atleast one key word in the sound data; storing in memory an associationbetween the key word and the facial image; receiving from the wearableimage sensor worn by a user another facial image of the individualduring a second interaction at a time other than during the time window;using image processing to determine that the individual in the firstinteraction is the individual in the second interaction; accessing thememory to locate the at least one key word from the first interaction;and during the second interaction, presenting on a display visible tothe user, the at least one key word, to thereby remind the user ofsubject matter of the first interaction.

In one embodiment, a method for retrieving and displaying key words fromprior conversations may be implemented by at least one processor of amobile communications device pairable with a wearable image sensor. Themethod may comprise receiving from the wearable image sensor worn by auser a facial image of an individual with whom the user interacted in afirst interaction during a time window; receiving sound data captured ina vicinity of the image sensor during the time window; processing thesound data to identify at least one key word in the sound data; storingin memory an association between the key word and the facial image;receiving from the wearable image sensor worn by a user another facialimage of the individual during a second interaction at a time other thanduring the time window; using image processing to determine that theindividual in the first interaction is the individual in the secondinteraction; accessing the memory to locate the at least one key wordfrom the first interaction; and during the second interaction,presenting on a display visible to the user, the at least one key word,to thereby remind the user of subject matter of the first interaction.

In one embodiment, a wearable apparatus may comprise a wearable imagesensor configured to capture a plurality of images from an environmentof a user of the wearable apparatus and at least one processor. The atleast one processor may be programmed to receive at least one image fromthe wearable image sensor; perform image processing of the at least oneimage to identify in the at least one image an environmental conditionof the user of the wearable apparatus; determine that the environmentalcondition is predetermined as a trigger for an action associated with amobile communications device wirelessly paired with the wearableapparatus; and cause the mobile communications device to trigger theaction in accordance with the environment condition and to thereby causethe action to occur via the mobile communications device.

In one embodiment, a non-transitory computer readable medium may containinstructions that may be executed by at least one processor. Theinstructions may cause the at least one processor to perform stepsincluding: receiving at least one image from a wearable image sensorworn by a user; performing image processing of the at least one image toidentify in the at least one image an environmental condition of theuser of the wearable apparatus; determining that the environmentalcondition is predetermined as a trigger for an action associated with amobile communications device wirelessly paired with the wearableapparatus; and causing the mobile communications device to trigger theaction in accordance with the environment condition and to thereby causethe action to occur via the mobile communications device.

In one embodiment, a wearable apparatus may comprise a wearable audioinput device configured to capture audio data from an environment of auser of the wearable apparatus and at least one processor. The at leastone processor may be programmed to receive audio data from the wearableaudio input device; perform audio processing of the audio data toidentify in the audio data an environmental condition of the user of thewearable apparatus; determine that the environmental condition ispredetermined as a trigger for an action associated with a mobilecommunications device wirelessly paired with the wearable apparatus; andcause the mobile communications device to trigger the action inaccordance with the environment condition and to thereby cause theaction to occur via the mobile communications device.

In an embodiment, a wearable apparatus may include a wearable audioinput device configured to capture audio data from an environment of auser of the wearable apparatus, a wearable image sensor configured tocapture a plurality of images from the environment of the user of thewearable apparatus, and at least one processor. The at least oneprocessor may be programmed to receive the audio data from the wearableaudio input device; receive at least one image from the wearable imagesensor; perform audio processing of the audio data and image processingof the at least one image to identify an environmental condition of theuser of the wearable apparatus; determine that the environmentalcondition is predetermined as a trigger for an action associated with amobile communications device wirelessly paired with the wearableapparatus and having an operational state; and cause the mobilecommunications device to trigger the action in accordance with theenvironment condition and to thereby cause the action to occur via themobile communications device.

In one embodiment, a wearable apparatus may comprise a wearable imagesensor configured to capture a plurality of images; and at least oneprocessor. The at least one processor may be programmed to receive, froma mobile communications device wirelessly paired with the wearableapparatus, a command to provide location information for the wearableapparatus; analyze, in response to the command, at least one imagedepicting an environment of the wearable apparatus to determine alocation of the wearable apparatus; and cause information about thelocation of the wearable apparatus to be transmitted to the mobilecommunications device.

In one embodiment, a non-transitory computer readable medium may containinstructions that when executed by at least one processor cause the atleast one processor to perform steps. The steps may include establishinga wireless pairing between a wearable image sensor and a mobilecommunications device; receiving, in a situation when the wearable imagesensor is misplaced, a user input for pinging the wearable image sensor;in response to the user input, wirelessly transmitting from the mobilecommunications device, a ping to the wearable image sensor, wherein theping includes an instruction for the wearable image sensor to capture atleast one image of surroundings of the image sensor; analyze the atleast one image to determine a location of the wearable apparatus; andcause, based on the location of the wearable apparatus, the locationinformation to be transmitted to the mobile communications device.

In one embodiment, a method for locating a wearable apparatus mayinclude receiving, from a mobile communications device wirelessly pairedwith the wearable apparatus, a command to provide location informationfor the wearable apparatus; analyzing, in response to the command, atleast one image depicting an environment of the wearable apparatus todetermine a location of the wearable apparatus; and causing informationabout the location of the wearable apparatus to be transmitted to themobile communications device.

In one embodiment, a wearable apparatus may include wearable imagesensor configured to capture a plurality of images and at least oneprocessor. The at least one processor may be programmed to: analyze atleast one image captured by the wearable image sensor; determine apositioning issue related to the wearable image sensor based on theanalysis of the at least one image; determine, in response to thepositioning issue, correction information for resolving the positioningissue; and cause the correction information to be provided to a user ofthe wearable apparatus.

In one embodiment, a non-transitory computer readable medium for use ina system employing a wearable image sensor pairable with a mobilecommunications device may contain instructions that when executed by atleast one processor cause the at least one processor to perform steps.The steps may include receiving a plurality of images from the wearableimage sensor; performing image processing on at least some of theimages; determining based on the image processing, an impediment toquality image capture; determining, in response to the determinedimpediment, correction information for resolving the positioning issue;and presenting the correction information on a display of the pairablemobile communications device.

In one embodiment, a method for correcting the positioning of a wearableimage sensor may include analyzing at least one image captured by thewearable image sensor; determining a positioning issue related to thewearable image sensor based on the analysis of the at least one image;determining, in response to the positioning issue, correctioninformation for resolving the positioning issue; and causing thecorrection information to be provided to a user of the wearableapparatus.

In one embodiment, a wearable apparatus may include a wearable imagesensor configured to capture a plurality of images and at least oneprocessor. The at least one processor may be programmed to analyze atleast one image captured by the wearable image sensor; determine apositioning issue related to the wearable image sensor based on theanalysis of the at least one image; determine, in response to thepositioning issue, correction information for resolving the positioningissue, wherein the correction information comprises executableinstructions configured to cause the at least one processor toautomatically resolve the positioning issue; and execute the correctioninformation to automatically resolve the positioning issue.

In one embodiment, a wearable apparatus may include a wearable imagesensor and at least one processor programmed to receive, from thewearable image sensor, a facial image of an individual with whom a userof the wearable apparatus is interacting; receive sound data capturedduring the interacting; process at least a portion of the sound data todetermine a spoken name of the individual; convert the spoken name totext; store, in memory, text associated with the spoken name in a mannerassociating the text with the facial image; after a subsequent encounterwith the individual, receive, from the wearable image sensor, asubsequent facial image of the individual; perform a look-up of anidentity of the individual based on the subsequent facial image;receive, from the memory the text of the spoken name of the individual;and cause a display in text of the name of the individual on a mobilecommunications device paired with the wearable apparatus.

In one embodiment, a non-transitory computer-readable medium for use ina system employing a wearable image sensor pairable with a mobilecommunications device may contain instructions that when executed by atleast one processor cause the at least one processor to perform steps.The steps may include receiving from a wearable image sensor worn by auser, a facial image of an individual with whom the user is interacting;receiving a sound file captured during the interacting; processing atleast a portion of the sound file to determine a spoken name of theindividual; converting the spoken name to text; storing in memory, textassociated with the spoken name in a manner associating the text withthe facial image; upon a subsequent encounter with the individual,receiving from the wearable image sensor a subsequent facial image ofthe individual; performing a look-up of an identity of the individualbased on the subsequent facial image; recalling from memory the text ofthe spoken name of the individual; and displaying in text on the mobilecommunications device of the user, the name of the individual.

In one embodiment, a wearable apparatus may include a wearable imagesensor configured to capture a plurality of images from an environmentof a user of the wearable apparatus, a memory unit configured to store aplurality of tasks, the plurality of tasks comprises at least a firsttask and a second task, and at least one processing device programmed toanalyze the plurality of images to obtain information related to atleast some of the plurality of tasks; based on the obtained information,assign a higher priority to the first task over the second task; andprovide feedback to the user based on the assigned priority.

In one embodiment, a method for capturing and processing images maycomprise receiving, from a wearable image sensor, a plurality of imagesfrom an environment of a user of a wearable apparatus; analyzing theplurality of images to obtain information related to at least some of aplurality of tasks, the plurality of tasks comprising at least a firsttask and a second task; based on the obtained information, assigning ahigher priority to the first task over the second task; and providingfeedback to the user based on the assigned priority.

In one embodiment, a wearable apparatus may analyze group dynamics. Thewearable apparatus may include a wearable image sensor configured tocapture a plurality of images from an environment of a user of thewearable apparatus. The wearable apparatus may also include at least oneprocessing device configured to perform a first analysis of theplurality of images to detect at least two persons, perform a secondanalysis of the plurality of images to determine association informationrelated to the at least two detected persons, and update a socialrepresentation based on the determined association information.

In one embodiment, a method may analyze group dynamics. The methodincludes obtaining, using a wearable image sensor, a plurality of imagescaptured from an environment of a user of a wearable apparatus,performing, using at least one processor of the wearable apparatus, afirst analysis of the plurality of images to detect at least twopersons, performing, using the at least one processor, a second analysisof the plurality of images to determine association information relatedto the at least two detected persons, and updating a socialrepresentation based on the determined association information.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processor and perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1A is a schematic illustration of an example of a user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1B is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1C is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 1D is a schematic illustration of an example of the user wearing awearable apparatus according to a disclosed embodiment.

FIG. 2 is a schematic illustration of an example system consistent withthe disclosed embodiments.

FIG. 3A is a schematic illustration of an example of the wearableapparatus shown in FIG. 1A.

FIG. 3B is an exploded view of the example of the wearable apparatusshown in FIG. 3A.

FIG. 4A is a schematic illustration of an example of the wearableapparatus shown in FIG. 1B from a first viewpoint.

FIG. 4B is a schematic illustration of the example of the wearableapparatus shown in FIG. 1B from a second viewpoint.

FIG. 5A is a block diagram illustrating an example of the components ofa wearable apparatus according to a first embodiment.

FIG. 5B is a block diagram illustrating an example of the components ofa wearable apparatus according to a second embodiment.

FIG. 5C is a block diagram illustrating an example of the components ofa wearable apparatus according to a third embodiment.

FIG. 6 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIG. 7 is a schematic illustration of an embodiment of a wearableapparatus including an orientable image capture unit.

FIG. 8 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 9 is a schematic illustration of a user wearing a wearableapparatus consistent with an embodiment of the present disclosure.

FIG. 10 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 11 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 12 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 13 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 14 is a schematic illustration of an embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure.

FIG. 15 is a schematic illustration of an embodiment of a wearableapparatus power unit including a power source.

FIG. 16 is a schematic illustration of an exemplary embodiment of awearable apparatus including protective circuitry.

FIG. 17 illustrates an exemplary embodiment of a memory containingsoftware modules for generating a face cloud consistent with the presentdisclosure.

FIG. 18A illustrates an example of a face cloud having prominence levelsindicated by size consistent with the present disclosure.

FIG. 18B illustrates an example of a face cloud having prominence levelsindicated by shape consistent with the present disclosure.

FIG. 18C illustrates an example of a face cloud including a linking wordconsistent with the present disclosure.

FIG. 18D illustrates an example of a face cloud including a plurality oflinking words consistent with the present disclosure.

FIG. 18E illustrates an example of a face cloud including a linkinggeographic location consistent with the present disclosure.

FIG. 18F illustrates an example of a face cloud including a linkingenvironmental characteristic consistent with the present disclosure.

FIG. 18G illustrates an example of a face cloud organized using atemporal indicator consistent with the present disclosure.

FIG. 18H illustrates an example of designation of a focal pointindividual that causes a rearrangement of the face cloud consistent withthe present disclosure.

FIG. 19A is a flowchart of an exemplary embodiment of a method forgenerating a visualization of image representations of a plurality ofpeople consistent with the present disclosure.

FIG. 19B is a flowchart of an exemplary embodiment of a method forgenerating a visual representation including a plurality of facialimages arranged based on at least one linking attribute consistent withthe present disclosure.

FIG. 19C is a flowchart of exemplary embodiment of a method forgenerating a subset of a plurality of facial images of individualshaving word lists with overlapping common words consistent with thepresent disclosure.

FIG. 20 illustrates an exemplary embodiment of a memory containingsoftware modules for indexing and searching people and objectsconsistent with the present disclosure.

FIG. 21A illustrates an example of a database for indexing facial imagesto linking attributes consistent with the present disclosure.

FIG. 21B illustrates an example of a database for indexing sound filesto extracted words and object images consistent with the presentdisclosure.

FIG. 21C illustrates an example of a database for indexing facial imagesto calendar dates consistent with the present disclosure.

FIG. 22A is a flowchart of an exemplary embodiment of a method forindexing facial images to linking attributes consistent with the presentdisclosure.

FIG. 22B is a flowchart of an exemplary embodiment of a method forindexing words in sound files to object images consistent with thepresent disclosure.

FIG. 22C is a flowchart of an exemplary embodiment of a method forindexing facial images to calendar dates consistent with the presentdisclosure.

FIG. 22D is a flowchart of an exemplary embodiment of a method forindexing facial images to contextual information extracted from imagesconsistent with the present disclosure.

FIG. 22E is a flowchart of an exemplary embodiment of a method forindexing visual context indicators extracted from images to contextualcategories associated with persons consistent with the presentdisclosure.

FIG. 23 illustrates an exemplary embodiment of a memory containingsoftware modules for automatic object comparison consistent with thepresent disclosure.

FIG. 24A illustrates an example of a trigger for performing a look up oftwo differing objects consistent with the present disclosure.

FIG. 24B illustrates another example of a trigger for performing a lookup of two differing objects consistent with the present disclosure.

FIG. 24C illustrates an example of a user interface for performing alook up of two differing objects consistent with the present disclosure.

FIG. 25 is a flowchart of an exemplary embodiment of a method forautomatic object comparison consistent with the present disclosure.

FIG. 26 illustrates an exemplary embodiment of a memory containingsoftware modules for retrieving and displaying key words from priorconversations, consistent with the present disclosure.

FIG. 27A illustrates an example of a database for indexing key words tointeraction frequencies, consistent with the present disclosure.

FIG. 27B illustrates another example of a database for indexing keywords to importance levels of individuals, consistent with the presentdisclosure.

FIG. 28 is a flowchart of an exemplary embodiment of a method forretrieving and displaying key words from prior conversations, consistentwith the present disclosure.

FIG. 29 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIGS. 30A and 30B are illustrations of exemplary environments detectedby a wearable apparatus consistent with the present disclosure.

FIG. 31A is a flowchart of a method for detecting a trigger in theenvironment of a user of the wearable apparatus according to a disclosedembodiment.

FIG. 31B is a flowchart of a method for detecting a trigger in theenvironment of a user of the wearable apparatus according to a disclosedembodiment.

FIG. 31C is a flowchart of a method for detecting a trigger in theenvironment of a user of the wearable apparatus according to a disclosedembodiment.

FIG. 32 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIGS. 33A, 33B, and 33C are illustrations of exemplary applications of awearable apparatus consistent with the present disclosure.

FIG. 34 is a flowchart of a method for providing location information toa user of a wearable apparatus according to a disclosed embodiment.

FIG. 35 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIGS. 36A, 36B, and 36C are illustrations of exemplary embodiments ofimage sensor occlusion consistent with the present disclosure.

FIG. 37 is a flowchart of a method for detecting and correcting ablockage of the image sensor of the wearable apparatus according to adisclosed embodiment.

FIG. 38 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIGS. 39A and 38B are flowcharts of methods for name tagging using awearable apparatus according to disclosed embodiments.

FIG. 40 shows example user interfaces according to disclosedembodiments.

FIG. 41 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure.

FIG. 42 is a flowchart of a method for prioritizing tasks according todisclosed embodiments.

FIGS. 43A and 43B show example captured images according to disclosedembodiments.

FIG. 43C shows example user interfaces according to disclosedembodiments.

FIG. 44 is an exemplary embodiment of a memory containing softwaremodules consistent with the present disclosure.

FIG. 45A is a schematic illustration for determining associationinformation based on a physical distance.

FIG. 45B is a schematic illustration for determining associationinformation based on a joint activity.

FIG. 45C is a schematic illustration for determining associationinformation based on a shared characteristic.

FIG. 45D is a schematic illustration for determining associationinformation based on a reaction to an action.

FIG. 46 is a schematic illustration of an update to a socialrepresentation based on a group dynamic.

FIG. 47 is a flowchart showing an exemplary process for analyzing agroup dynamic.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

FIG. 1A illustrates a user 100 wearing an apparatus 110 that isphysically connected (or integral) to glasses 130, consistent with thedisclosed embodiments. Glasses 130 may be prescription glasses,magnifying glasses, non-prescription glasses, safety glasses,sunglasses, etc. Additionally, in some embodiments, glasses 130 mayinclude parts of a frame and earpieces, nosepieces, etc., and one or nolenses. Thus, in some embodiments, glasses 130 may function primarily tosupport apparatus 110, and/or an augmented reality display device orother optical display device. In some embodiments, apparatus 110 mayinclude an image sensor (not shown in FIG. 1A) for capturing real-timeimage data of the field-of-view of user 100. The term “image data”includes any form of data retrieved from optical signals in thenear-infrared, infrared, visible, and ultraviolet spectrums. The imagedata may include video clips and/or photographs. In some embodiments,the image sensor of apparatus 110 (not shown) may have a field-of-viewthat is substantially similar (e.g., at least 50% overlapping, at least70% overlapping, at least 90% overlapping, or the like) to the field-ofview of user 100.

In some embodiments, a magnet or other device may be embedded withinglasses 130, for example within a handle of the glasses, such that noparticular device or installation is required for attaching apparatus110 to glasses 130. The same apparatus can then be moved betweendifferent glasses of the user.

In some embodiments, apparatus 110 may communicate wirelessly or via awire with a computing device 120. In some embodiments, computing device120 may include, for example, a smartphone, or a tablet, or a dedicatedprocessing unit, which may be portable (e.g., can be carried in a pocketof user 100). Although shown in Fig. lA as an external device, in someembodiments, computing device 120 may be provided as part of wearableapparatus 110 or glasses 130, whether integral thereto or mountedthereon. In some embodiments, computing device 120 may be included in anaugmented reality display device or optical head mounted displayprovided integrally or mounted to glasses 130. In other embodiments, forexample, when computing device 120 is a smartphone, computing device 120may include a display device which may be used as the augmented realitydisplay device or other optical display device. In other embodiments,computing device 120 may be provided as part of another wearable orportable apparatus of user 100 including a wrist-strap, amultifunctional watch, a button, a clip-on, etc. In other embodiments,computing device 120 may be provided as part of another system, such asan on-board automobile computing or navigation system. A person skilledin the art can appreciate that different types of computing devices andarrangements of devices may implement the functionality of the disclosedembodiments. Accordingly, in other implementations, computing device 120may include a Personal Computer (PC), laptop, an Internet server, etc.

FIG. 1B illustrates user 100 wearing apparatus 110 that is physicallyconnected to a necklace 140, consistent with a disclosed embodiment.Such a configuration of apparatus 110 may be suitable for users that donot wear glasses some or all of the time. In this embodiment, user 100can easily wear apparatus 110, and take it off.

FIG. 1C illustrates user 100 wearing apparatus 110 that is physicallyconnected to a belt 150, consistent with a disclosed embodiment. Such aconfiguration of apparatus 110 may be designed as a belt buckle.Alternatively, apparatus 110 may include a clip for attaching to variousclothing articles, such as belt 150, a vest, a pocket, a collar, a capor hat or other portion of a clothing article.

FIG. 1D illustrates user 100 wearing apparatus 110 that is physicallyconnected to a wrist strap 160, consistent with a disclosed embodiment.Although the aiming direction of apparatus 110, according to thisembodiment, may not match the field-of-view of user 100, apparatus 110may include the ability to identify a hand-related trigger based on thetracked eye movement of a user 100 indicating that user 100 is lookingin the direction of the wrist strap 160. Wrist strap 160 may alsoinclude an accelerometer, a gyroscope, or other sensor for determiningmovement or orientation of a user's 100 hand for identifying ahand-related trigger.

FIG. 2 is a schematic illustration of an exemplary system 200 includinga wearable apparatus 110, worn by user 100, and an optional computingdevice 120 and/or a server 250 capable of communicating with apparatus110 via a network 240, consistent with disclosed embodiments. In someembodiments, apparatus 110 may capture and analyze image data, identifya hand-related trigger present in the image data, and perform an actionand/or provide feedback to a user 100, based at least in part on theidentification of the hand-related trigger. In some embodiments,optional computing device 120 and/or server 250 may provide additionalfunctionality to enhance interactions of user 100 with his or herenvironment, as described in greater detail below.

According to the disclosed embodiments, apparatus 110 may include animage sensor system 220 for capturing real-time image data of thefield-of-view of user 100. In some embodiments, apparatus 110 may alsoinclude a processing unit 210 for controlling and performing thedisclosed functionality of apparatus 110, such as to control the captureof image data, analyze the image data, and perform an action and/oroutput a feedback based on a hand-related trigger identified in theimage data. According to the disclosed embodiments, a hand-relatedtrigger may include a gesture performed by user 100 involving a portionof a hand of user 100. Further, consistent with some embodiments, ahand-related trigger may include a wrist-related trigger. Additionally,in some embodiments, apparatus 110 may include a feedback outputtingunit 230 for producing an output of information to user 100.

As discussed above, apparatus 110 may include an image sensor 220 forcapturing image data. The term “image sensor” refers to a device capableof detecting and converting optical signals in the near-infrared,infrared, visible, and ultraviolet spectrums into electrical signals.The electrical signals may be used to form an image or a video stream(i.e. image data) based on the detected signal. The term “image data”includes any form of data retrieved from optical signals in thenear-infrared, infrared, visible, and ultraviolet spectrums. Examples ofimage sensors may include semiconductor charge-coupled devices (CCD),active pixel sensors in complementary metal-oxide-semiconductor (CMOS),or N-type metal-oxide-semiconductor (NMOS, Live MOS). In some cases,image sensor 220 may be part of a camera included in apparatus 110.

Apparatus 110 may also include a processor 210 for controlling imagesensor 220 to capture image data and for analyzing the image dataaccording to the disclosed embodiments. As discussed in further detailbelow with respect to FIG. 5A, processor 210 may include a “processingdevice” for performing logic operations on one or more inputs of imagedata and other data according to stored or accessible softwareinstructions providing desired functionality. In some embodiments,processor 210 may also control feedback outputting unit 230 to providefeedback to user 100 including information based on the analyzed imagedata and the stored software instructions. As the term is used herein, a“processing device” may access memory where executable instructions arestored or, in some embodiments, a “processing device” itself may includeexecutable instructions (e.g., stored in memory included in theprocessing device).

In some embodiments, the information or feedback information provided touser 100 may include time information. The time information may includeany information related to a current time of day and, as describedfurther below, may be presented in any sensory perceptive manner. Insome embodiments, time information may include a current time of day ina preconfigured format (e.g., 2:30 pm or 14:30). Time information mayinclude the time in the user's current time zone (e.g., based on adetermined location of user 100), as well as an indication of the timezone and/or a time of day in another desired location. In someembodiments, time information may include a number of hours or minutesrelative to one or more predetermined times of day. For example, in someembodiments, time information may include an indication that three hoursand fifteen minutes remain until a particular hour (e.g., until 6:00pm), or some other predetermined time. Time information may also includea duration of time passed since the beginning of a particular activity,such as the start of a meeting or the start of a jog, or any otheractivity. In some embodiments, the activity may be determined based onanalyzed image data. In other embodiments, time information may alsoinclude additional information related to a current time and one or moreother routine, periodic, or scheduled events. For example, timeinformation may include an indication of the number of minutes remaininguntil the next scheduled event, as may be determined from a calendarfunction or other information retrieved from computing device 120 orserver 250, as discussed in further detail below.

Feedback outputting unit 230 may include one or more feedback systemsfor providing the output of information to user 100. In the disclosedembodiments, the audible or visual feedback may be provided via any typeof connected audible or visual system or both. Feedback of informationaccording to the disclosed embodiments may include audible feedback touser 100 (e.g., using a BluetoothTM or other wired or wirelesslyconnected speaker, or a bone conduction headphone). Feedback outputtingunit 230 of some embodiments may additionally or alternatively produce avisible output of information to user 100, for example, as part of anaugmented reality display projected onto a lens of glasses 130 orprovided via a separate heads up display in communication with apparatus110, such as a display 260 provided as part of computing device 120,which may include an onboard automobile heads up display, an augmentedreality device, a virtual reality device, a smartphone, PC, table, etc..

The term “computing device” refers to a device including a processingunit and having computing capabilities. Some examples of computingdevice 120 include a PC, laptop, tablet, or other computing systems suchas an on-board computing system of an automobile, for example, eachconfigured to communicate directly with apparatus 110 or server 250 overnetwork 240. Another example of computing device 120 includes asmartphone having a display 260. In some embodiments, computing device120 may be a computing system configured particularly for apparatus 110,and may be provided integral to apparatus 110 or tethered thereto.Apparatus 110 can also connect to computing device 120 over network 240via any known wireless standard (e.g., Wi-Fi, Bluetooth®, etc.), as wellas near-field capacitive coupling, and other short range wirelesstechniques, or via a wired connection. In an embodiment in whichcomputing device 120 is a smartphone, computing device 120 may have adedicated application installed therein. For example, user 100 may viewon display 260 data (e.g., images, video clips, extracted information,feedback information, etc.) that originates from or is triggered byapparatus 110. In addition, user 100 may select part of the data forstorage in server 250.

Network 240 may be a shared, public, or private network, may encompass awide area or local area, and may be implemented through any suitablecombination of wired and/or wireless communication networks. Network 240may further comprise an intranet or the Internet. In some embodiments,network 240 may include short range or near-field wireless communicationsystems for enabling communication between apparatus 110 and computingdevice 120 provided in close proximity to each other, such as on or neara user's body, for example. Apparatus 110 may establish a connection tonetwork 240 autonomously, for example, using a wireless module (e.g.,Wi-Fi, cellular). In some embodiments, apparatus 110 may use thewireless module when being connected to an external power source, toprolong battery life. Further, communication between apparatus 110 andserver 250 may be accomplished through any suitable communicationchannels, such as, for example, a telephone network, an extranet, anintranet, the Internet, satellite communications, off-linecommunications, wireless communications, transponder communications, alocal area network (LAN), a wide area network (WAN), and a virtualprivate network (VPN).

As shown in FIG. 2, apparatus 110 may transfer or receive data to/fromserver 250 via network 240. In the disclosed embodiments, the data beingreceived from server 250 and/or computing device 120 may includenumerous different types of information based on the analyzed imagedata, including information related to a commercial product, or aperson's identity, an identified landmark, and any other informationcapable of being stored in or accessed by server 250. In someembodiments, data may be received and transferred via computing device120. Server 250 and/or computing device 120 may retrieve informationfrom different data sources (e.g., a user specific database or a user'ssocial network account or other account, the Internet, and other managedor accessible databases) and provide information to apparatus 110related to the analyzed image data and a recognized trigger according tothe disclosed embodiments. In some embodiments, calendar-relatedinformation retrieved from the different data sources may be analyzed toprovide certain time information or a time-based context for providingcertain information based on the analyzed image data.

An example of wearable apparatus 110 incorporated with glasses 130according to some embodiments (as discussed in connection with FIG. 1A)is shown in greater detail in FIG. 3A. In some embodiments, apparatus110 may be associated with a structure (not shown in FIG. 3A) thatenables easy detaching and reattaching of apparatus 110 to glasses 130.In some embodiments, when apparatus 110 attaches to glasses 130, imagesensor 220 acquires a set aiming direction without the need fordirectional calibration. The set aiming direction of image sensor 220may substantially coincide with the field-of-view of user 100. Forexample, a camera associated with image sensor 220 may be installedwithin apparatus 110 in a predetermined angle in a position facingslightly downwards (e.g., 5-15 degrees from the horizon). Accordingly,the set aiming direction of image sensor 220 may substantially match thefield-of-view of user 100.

FIG. 3B is an exploded view of the components of the embodimentdiscussed regarding FIG. 3A. Attaching apparatus 110 to glasses 130 maytake place in the following way. Initially, a support 310 may be mountedon glasses 130 using a screw 320, in the side of support 310. Then,apparatus 110 may be clipped on support 310 such that it is aligned withthe field-of-view of user 100. The term “support” includes any device orstructure that enables detaching and reattaching of a device including acamera to a pair of glasses or to another object (e.g., a helmet).Support 310 may be made from plastic (e.g., polycarbonate), metal (e.g.,aluminum), or a combination of plastic and metal (e.g., carbon fibergraphite). Support 310 may be mounted on any kind of glasses (e.g.,eyeglasses, sunglasses, 3D glasses, safety glasses, etc.) using screws,bolts, snaps, or any fastening means used in the art.

In some embodiments, support 310 may include a quick release mechanismfor disengaging and reengaging apparatus 110. For example, support 310and apparatus 110 may include magnetic elements. As an alternativeexample, support 310 may include a male latch member and apparatus 110may include a female receptacle. In other embodiments, support 310 canbe an integral part of a pair of glasses, or sold separately andinstalled by an optometrist. For example, support 310 may be configuredfor mounting on the arms of glasses 130 near the frame front, but beforethe hinge. Alternatively, support 310 may be configured for mounting onthe bridge of glasses 130. As described above with reference to FIG. 1A,in some embodiments, glasses 130 may have a magnet embedded therein, forexample within a handle, such that apparatus 110 can be immediatelyattached or detached from glasses 130 without further structures orinstallations, and such that no structure is left attached to glasses130 when apparatus 110 is detached.

In some embodiments, apparatus 110 may be provided as part of a glassesframe 130, with or without lenses. Additionally, in some embodiments,apparatus 110 may be configured to provide an augmented reality displayprojected onto a lens of glasses 130 (if provided), or alternatively,may include a display for projecting information, for example, accordingto the disclosed embodiments. Apparatus 110 may include the additionaldisplay or alternatively, may be in communication with a separatelyprovided display system that may or may not be attached to glasses 130.

In some embodiments, apparatus 110 may be implemented in a form otherthan wearable glasses, as described above with respect to FIGS. 1B-1D,for example. FIG. 4A is a schematic illustration of an example of anadditional embodiment of apparatus 110 from a first viewpoint. Theviewpoint shown in FIG. 4A is from the front of apparatus 110. Apparatus110 includes an image sensor 220, a clip (not shown), a function button(not shown) and a hanging ring 410 for attaching apparatus 110 to, forexample, necklace 140, as shown in FIG. 1B. When apparatus 110 hangs onnecklace 140, the aiming direction of image sensor 220 may not fullycoincide with the field-of-view of user 100, but the aiming directionwould still correlate with the field-of-view of user 100.

FIG. 4B is a schematic illustration of the example of a secondembodiment of apparatus 110, from a second viewpoint. The viewpointshown in FIG. 4B is from a side orientation of apparatus 110. Inaddition to hanging ring 410, as shown in FIG. 4B, apparatus 110 mayfurther include a clip 420. User 100 can use clip 420 to attachapparatus 110 to a shirt or belt 150, as illustrated in FIG. 1C. Clip420 may provide an easy mechanism for disengaging and reengagingapparatus 110 from different articles of clothing. In other embodiments,apparatus 110 may include a female receptacle for connecting with a malelatch of a car mount or universal stand.

In some embodiments, apparatus 110 includes a function button 430 forenabling user 100 to provide input to apparatus 110. Function button 430may accept different types of tactile input (e.g., a tap, a click, adouble-click, a long press, a right-to-left slide, a left-to-rightslide). In some embodiments, each type of input may be associated with adifferent action. For example, a tap may be associated with the functionof taking a picture, while a right-to-left slide may be associated withthe function of recording a video.

The example embodiments discussed above with respect to FIGS. 3A, 3B,4A, and 4B are not limiting. In some embodiments, apparatus 110 may beimplemented in any suitable configuration for performing the disclosedmethods. For example, referring back to FIG. 2, the disclosedembodiments may implement an apparatus 110 according to anyconfiguration including an image sensor 220 and a processor unit 210 toperform image analysis and for communicating with a feedback unit 230.

FIG. 5A is a block diagram illustrating the components of apparatus 110according to an example embodiment. As shown in FIG. 5A, and assimilarly discussed above, apparatus 110 includes an image sensor 220, amemory 550, a processor 210, a feedback outputting unit 230, a wirelesstransceiver 530, and a mobile power source 520. In other embodiments,apparatus 110 may also include buttons, other sensors such as amicrophone, and/or inertial measurements devices such as accelerometers,gyroscopes, magnetometers, temperature sensors, color sensors, lightsensors, etc. Apparatus 110 may further include a data port 570 and apower connection 510 with suitable interfaces for connecting with anexternal power source or an external device (not shown).

Processor 210, depicted in FIG. 5A, may include any suitable processingdevice. The term “processing device” includes any physical device havingan electric circuit that performs a logic operation on input or inputs.For example, processing device may include one or more integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field-programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. The instructions executed by the processing device may, forexample, be pre-loaded into a memory integrated with or embedded intothe processing device or may be stored in a separate memory (e.g.,memory 550). Memory 550 may comprise a Random Access Memory (RAM), aRead-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium,a flash memory, other permanent, fixed, or volatile memory, or any othermechanism capable of storing instructions.

Although, in the embodiment illustrated in FIG. 5A, apparatus 110includes one processing device (e.g., processor 210), apparatus 110 mayinclude more than one processing device. Each processing device may havea similar construction, or the processing devices may be of differingconstructions that are electrically connected or disconnected from eachother. For example, the processing devices may be separate circuits orintegrated in a single circuit. When more than one processing device isused, the processing devices may be configured to operate independentlyor collaboratively. The processing devices may be coupled electrically,magnetically, optically, acoustically, mechanically or by other meansthat permit them to interact.

In some embodiments, processor 210 may process a plurality of imagescaptured from the environment of user 100 to determine differentparameters related to capturing subsequent images. For example,processor 210 can determine, based on information derived from capturedimage data, a value for at least one of the following: an imageresolution, a compression ratio, a cropping parameter, frame rate, afocus point, an exposure time, an aperture size, and a lightsensitivity. Additionally or alternatively, one or more of the valuesmay be received from image sensor 220. The determined or received valuemay be used in capturing at least one subsequent image. Additionally,processor 210 can detect images including at least one hand-relatedtrigger in the environment of the user and perform an action and/orprovide an output of information to a user via feedback outputting unit230.

In another embodiment, processor 210 can change the aiming direction ofimage sensor 220. For example, when apparatus 110 is attached with clip420, the aiming direction of image sensor 220 may not coincide with thefield-of-view of user 100. Processor 210 may recognize certainsituations from the analyzed image data and adjust the aiming directionof image sensor 220 to capture relevant image data. For example, in oneembodiment, processor 210 may detect an interaction with anotherindividual and sense that the individual is not fully in view, becauseimage sensor 220 is tilted down. Responsive thereto, processor 210 mayadjust the aiming direction of image sensor 220 to capture image data ofthe individual. Other scenarios are also contemplated where processor210 may recognize the need to adjust an aiming direction of image sensor220.

In some embodiments, processor 210 may communicate data tofeedback-outputting unit 230, which may include any device configured toprovide information to a user 100. Feedback outputting unit 230 may beprovided as part of apparatus 110 (as shown) or may be provided externalto apparatus 110 and communicatively coupled thereto.Feedback-outputting unit 230 may be configured to output visual ornonvisual feedback based on signals received from processor 210, such aswhen processor 210 recognizes a hand-related trigger in the analyzedimage data.

The term “feedback” refers to any output or information provided inresponse to processing at least one image in an environment. In someembodiments, as similarly described above, feedback may include anaudible or visible indication of time information, detected text ornumerals, the value of currency, a branded product, a type of food ordrink, a person's identity, the identity of a landmark or otherenvironmental situation or condition including the street names at anintersection or the color of a traffic light, etc., a machine or device,as well as other information associated with each of these. For example,in some embodiments, feedback may include additional informationregarding the amount of currency still needed to complete a transaction,information regarding the identified person, historical information ortimes and prices of admission etc. of a detected landmark etc. In someembodiments, feedback may include an audible tone, a tactile response,and/or information previously recorded by user 100. Feedback-outputtingunit 230 may comprise appropriate components for outputting acousticaland tactile feedback. For example, feedback-outputting unit 230 maycomprise audio headphones, a hearing aid type device, a speaker, a boneconduction headphone, interfaces that provide tactile cues, vibrotactilestimulators, etc. In some embodiments, processor 210 may communicatesignals with an external feedback outputting unit 230 via a wirelesstransceiver 530, a wired connection, or some other communicationinterface. In some embodiments, feedback outputting unit 230 may alsoinclude any suitable display device for visually displaying informationto user 100.

As shown in FIG. 5A, apparatus 110 includes memory 550. Memory 550 maystore one or more sets of instructions accessible to processor 210 toperform the disclosed methods, including instructions for recognizing ahand-related trigger in the image data. In some embodiments memory 550may store image data (e.g., images, videos) captured from theenvironment of user 100. In addition, memory 550 may store informationspecific to user 100, such as image representations of knownindividuals, favorite products, personal items, and calendar orappointment information, etc. In some embodiments, processor 210 maydetermine, for example, which type of image data to store based onavailable storage space in memory 550. In another embodiment, processor210 may extract information from the image data stored in memory 550.

As further shown in FIG. 5A, apparatus 110 includes mobile power source520. The term “mobile power source” includes any device capable ofproviding electrical power, which can be easily carried by hand (e.g.,mobile power source 520 may weigh less than a pound). The mobility ofthe power source enables user 100 to use apparatus 110 in a variety ofsituations. In some embodiments, mobile power source 520 may include oneor more batteries (e.g., nickel-cadmium batteries, nickel-metal hydridebatteries, and lithium-ion batteries) or any other type of electricalpower supply. In other embodiments, mobile power source 520 may berechargeable and contained within a casing that holds apparatus 110. Inyet other embodiments, mobile power source 520 may include one or moreenergy harvesting devices for converting ambient energy into electricalenergy (e.g., portable solar power units, human vibration units, etc.).

Mobile power source 520 may power one or more wireless transceivers(e.g., wireless transceiver 530 in FIG. 5A). The term “wirelesstransceiver” refers to any device configured to exchange transmissionsover an air interface by use of radio frequency, infrared frequency,magnetic field, or electric field. Wireless transceiver 530 may use anyknown standard to transmit and/or receive data (e.g., Wi-Fi, Bluetooth®,Bluetooth Smart, 802.15.4, or ZigBee). In some embodiments, wirelesstransceiver 530 may transmit data (e.g., raw image data, processed imagedata, extracted information) from apparatus 110 to computing device 120and/or server 250. Wireless transceiver 530 may also receive data fromcomputing device 120 and/or server 250. In other embodiments, wirelesstransceiver 530 may transmit data and instructions to an externalfeedback outputting unit 230.

FIG. 5B is a block diagram illustrating the components of apparatus 110according to another example embodiment. In some embodiments, apparatus110 includes a first image sensor 220 a, a second image sensor 220 b, amemory 550, a first processor 210 a, a second processor 210 b, afeedback outputting unit 230, a wireless transceiver 530, a mobile powersource 520, and a power connector 510. In the arrangement shown in FIG.5B, each of the image sensors may provide images in different parameterssuch as image resolution, or face a different direction. Alternatively,each image sensor may be associated with a different type of camera(e.g., a wide angle camera, a narrow angle camera, an IR camera, etc.).In some embodiments, apparatus 110 can select which image sensor to usebased on various factors. For example, processor 210 a may determine,based on available storage space in memory 550, to capture subsequentimages in a certain resolution.

Apparatus 110 may operate in a first processing-mode and in a secondprocessing-mode, such that the first processing-mode may consume lesspower than the second processing-mode. For example, in the firstprocessing-mode, apparatus 110 may capture images and process thecaptured images to make real-time decisions based on an identifiedhand-related trigger, for example. In the second processing-mode,apparatus 110 may extract information from stored images in memory 550and delete images from memory 550. In some embodiments, mobile powersource 520 may provide more than fifteen hours of processing in thefirst processing-mode and about three hours of processing in the secondprocessing-mode. Accordingly, different processing-modes may allowmobile power source 520 to produce sufficient power for poweringapparatus 110 for various time periods (e.g., more than two hours, morethan four hours, more than ten hours, etc.).

In some embodiments, apparatus 110 may use first processor 210 a in thefirst processing-mode when powered by mobile power source 520, andsecond processor 210 b in the second processing-mode when powered byexternal power source 580 that is connectable via power connector 510.In other embodiments, apparatus 110 may determine, based on predefinedconditions, which processors or which processing modes to use. Apparatus110 may operate in the second processing-mode even when apparatus 110 isnot powered by external power source 580. For example, apparatus 110 maydetermine that it should operate in the second processing-mode whenapparatus 110 is not powered by external power source 580, if theavailable storage space in memory 550 for storing new image data islower than a predefined threshold. Accordingly, apparatus 110 mayextract information from stored images in memory 550 and delete imagesfrom memory 550 even if apparatus 110 is not powered by external powersource 580 when memory 550 has free space not exceeding the predefinedthreshold.

Although one wireless transceiver is depicted in FIG. 5B, apparatus 110may include more than one wireless transceiver (e.g., two wirelesstransceivers). In an arrangement with more than one wirelesstransceiver, each of the wireless transceivers may use a differentstandard to transmit and/or receive data. In some embodiments, a firstwireless transceiver may communicate with server 250 or computing device120 using a cellular standard (e.g., LTE or GSM), and a second wirelesstransceiver may communicate with server 250 or computing device 120using a short-range standard (e.g., Wi-Fi or Bluetooth®). In someembodiments, apparatus 110 may use the first wireless transceiver whenthe wearable apparatus is powered by a mobile power source included inthe wearable apparatus, and use the second wireless transceiver when thewearable apparatus is powered by an external power source.

FIG. 5C is a block diagram illustrating the components of apparatus 110according to another example embodiment including computing device 120.In this embodiment, apparatus 110 includes an image sensor 220, a memory550 a, a first processor 210, a feedback-outputting unit 230, a wirelesstransceiver 530 a, a mobile power source 520, and a power connector 510.As further shown in FIG. 5C, computing device 120 includes a processor540, a feedback-outputting unit 545, a memory 550 b, a wirelesstransceiver 530 b, and a display 260. One example of computing device120 is a smartphone or tablet having a dedicated application installedtherein. In other embodiments, computing device 120 may include anyconfiguration such as an on-board automobile computing system, a PC, alaptop, and any other system consistent with the disclosed embodiments.In this example, user 100 may view feedback output in response toidentification of a hand-related trigger on display 260. Additionally,user 100 may view other data (e.g., images, video clips, objectinformation, schedule information, extracted information, etc.) ondisplay 260. In addition, user 100 may communicate with server 250 viacomputing device 120.

In some embodiments, processor 210 and processor 540 are configured toextract information from captured image data. The term “extractinginformation” includes any process by which information associated withobjects, individuals, locations, events, etc., is identified in thecaptured image data by any means known to those of ordinary skill in theart. In some embodiments, apparatus 110 may use the extractedinformation to send feedback or other real-time indications to feedbackoutputting unit 230 or to computing device 120. In some embodiments,processor 210 may identify in the image data the individual standing infront of user 100, and send computing device 120 the name of theindividual and the last time user 100 met the individual. In anotherembodiment, processor 210 may identify in the image data, one or morevisible triggers, including a hand-related trigger, and determinewhether the trigger is associated with a person other than the user ofthe wearable apparatus, to selectively determine whether to perform anaction associated with the trigger. One such action may be to provide afeedback to user 100 via feedback-outputting unit 230 provided as partof (or in communication with) apparatus 110 or via a feedback unit 545provided as part of computing device 120. For example,feedback-outputting unit 545 may be in communication with display 260 tocause the display 260 to visibly output information. In someembodiments, processor 210 may identify in the image data a hand-relatedtrigger and send computing device 120 an indication of the trigger.Processor 540 may then process the received trigger information andprovide an output via feedback outputting unit 545 or display 260 basedon the hand-related trigger. In other embodiments, processor 540 maydetermine a hand-related trigger and provide suitable feedback similarto the above, based on image data received from apparatus 110. In someembodiments, processor 540 may provide instructions or otherinformation, such as environmental information to apparatus 110 based onan identified hand-related trigger.

In some embodiments, processor 210 may identify other environmentalinformation in the analyzed images, such as an individual standing infront user 100, and send computing device 120 information related to theanalyzed information such as the name of the individual and the lasttime user 100 met the individual. In a different embodiment, processor540 may extract statistical information from captured image data andforward the statistical information to server 250. For example, certaininformation regarding the types of items a user purchases, or thefrequency a user patronizes a particular merchant, etc. may bedetermined by processor 540. Based on this information, server 250 maysend computing device 120 coupons and discounts associated with theuser's preferences.

When apparatus 110 is connected or wirelessly connected to computingdevice 120, apparatus 110 may transmit at least part of the image datastored in memory 550 a for storage in memory 550 b. In some embodiments,after computing device 120 confirms that transferring the part of imagedata was successful, processor 210 may delete the part of the imagedata. The term “delete” means that the image is marked as ‘deleted’ andother image data may be stored instead of it, but does not necessarilymean that the image data was physically removed from the memory.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the disclosed embodiments. Not all components are essential for theoperation of apparatus 110. Any component may be located in anyappropriate apparatus and the components may be rearranged into avariety of configurations while providing the functionality of thedisclosed embodiments. For example, in some embodiments, apparatus 110may include a camera, a processor, and a wireless transceiver forsending data to another device. Therefore, the foregoing configurationsare examples and, regardless of the configurations discussed above,apparatus 110 can capture, store, and/or process images.

Further, the foregoing and following description refers to storingand/or processing images or image data. In the embodiments disclosedherein, the stored and/or processed images or image data may comprise arepresentation of one or more images captured by image sensor 220. Asthe term is used herein, a “representation” of an image (or image data)may include an entire image or a portion of an image. A representationof an image (or image data) may have the same resolution or a lowerresolution as the image (or image data), and/or a representation of animage (or image data) may be altered in some respect (e.g., becompressed, have a lower resolution, have one or more colors that arealtered, etc.).

For example, apparatus 110 may capture an image and store arepresentation of the image that is compressed, for example, as a .JPGfile. As another example, apparatus 110 may capture an image in color,but store a black-and-white representation of the color image. As yetanother example, apparatus 110 may capture an image and store adifferent representation of the image (e.g., a portion of the image).For example, apparatus 110 may store a portion of an image that includesa face of a person who appears in the image, but that does notsubstantially include the environment surrounding the person. Similarly,apparatus 110 may, for example, store a portion of an image thatincludes a product that appears in the image, but does not substantiallyinclude the environment surrounding the product. As yet another example,apparatus 110 may store a representation of an image at a reducedresolution (i.e., at a resolution that is of a lower value than that ofthe captured image). Storing representations of images may allowapparatus 110 to save storage space in memory 550. Furthermore,processing representations of images may allow apparatus 110 to improveprocessing efficiency and/or help to preserve battery life.

In addition to the above, in some embodiments, any one of apparatus 110or computing device 120, via processor 210 or 540, may further processthe captured image data to provide additional functionality to recognizeobjects and/or gestures and/or other information in the captured imagedata. In some embodiments, actions may be taken based on the identifiedobjects, gestures, or other information. In some embodiments, processor210 or 540 may identify in the image data, one or more visible triggers,including a hand-related trigger, and determine whether the trigger isassociated with a person other than the user to determine whether toperform an action associated with the trigger.

Some embodiments of the present disclosure may include an apparatussecurable to an article of clothing of a user. Such an apparatus mayinclude two portions, connectable by a connector. A capturing unit maybe designed to be worn on the outside of a user's clothing, and mayinclude an image sensor for capturing images of a user's environment.The capturing unit may be connected to or connectable to a power unit,which may be configured to house a power source and a processing device.

The capturing unit may be a small device including a camera or otherdevice for capturing images. The capturing unit may be designed to beinconspicuous and unobtrusive, and may be configured to communicate witha power unit concealed by a user's clothing. The power unit may includebulkier aspects of the system, such as transceiver antennas, at leastone battery, a processing device, etc. In some embodiments,communication between the capturing unit and the power unit may beprovided by a data cable included in the connector, while in otherembodiments, communication may be wirelessly achieved between thecapturing unit and the power unit. Some embodiments may permitalteration of the orientation of an image sensor of the capture unit,for example to better capture images of interest.

FIG. 6 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure. Included inmemory 550 are orientation identification module 601, orientationadjustment module 602, and monitoring module 603. Modules 601, 602, 603may contain software instructions for execution by at least oneprocessing device, e.g., processor 210, included with a wearableapparatus. Orientation identification module 601, orientation adjustmentmodule 602, and monitoring module 603 may cooperate to provideorientation adjustment for a capturing unit incorporated into wirelessapparatus 110.

FIG. 7 illustrates an exemplary capturing unit 710 including anorientation adjustment unit 705. Orientation adjustment unit 705 may beconfigured to permit the adjustment of image sensor 220. As illustratedin FIG. 7, orientation adjustment unit 705 may include an eye-ball typeadjustment mechanism. In alternative embodiments, orientation adjustmentunit 705 may include gimbals, adjustable stalks, pivotable mounts, andany other suitable unit for adjusting an orientation of image sensor220.

Image sensor 220 may be configured to be movable with the head of user100 in such a manner that an aiming direction of image sensor 220substantially coincides with a field of view of user 100. For example,as described above, a camera associated with image sensor 220 may beinstalled within capturing unit 710 at a predetermined angle in aposition facing slightly upwards or downwards, depending on an intendedlocation of capturing unit 710. Accordingly, the set aiming direction ofimage sensor 220 may match the field-of-view of user 100. In someembodiments, processor 210 may change the orientation of image sensor220 using image data provided from image sensor 220. For example,processor 210 may recognize that a user is reading a book and determinethat the aiming direction of image sensor 220 is offset from the text.That is, because the words in the beginning of each line of text are notfully in view, processor 210 may determine that image sensor 220 istilted in the wrong direction. Responsive thereto, processor 210 mayadjust the aiming direction of image sensor 220.

Orientation identification module 601 may be configured to identify anorientation of an image sensor 220 of capturing unit 710. An orientationof an image sensor 220 may be identified, for example, by analysis ofimages captured by image sensor 220 of capturing unit 710, by tilt orattitude sensing devices within capturing unit 710, and by measuring arelative direction of orientation adjustment unit 705 with respect tothe remainder of capturing unit 710.

Orientation adjustment module 602 may be configured to adjust anorientation of image sensor 220 of capturing unit 710. As discussedabove, image sensor 220 may be mounted on an orientation adjustment unit705 configured for movement. Orientation adjustment unit 705 may beconfigured for rotational and/or lateral movement in response tocommands from orientation adjustment module 602. In some embodimentsorientation adjustment unit 705 may be adjust an orientation of imagesensor 220 via motors, electromagnets, permanent magnets, and/or anysuitable combination thereof.

In some embodiments, monitoring module 603 may be provided forcontinuous monitoring. Such continuous monitoring may include tracking amovement of at least a portion of an object included in one or moreimages captured by the image sensor. For example, in one embodiment,apparatus 110 may track an object as long as the object remainssubstantially within the field-of-view of image sensor 220. Inadditional embodiments, monitoring module 603 may engage orientationadjustment module 602 to instruct orientation adjustment unit 705 tocontinually orient image sensor 220 towards an object of interest. Forexample, in one embodiment, monitoring module 603 may cause image sensor220 to adjust an orientation to ensure that a certain designated object,for example, the face of a particular person, remains within thefield-of view of image sensor 220, even as that designated object movesabout. In another embodiment, monitoring module 603 may continuouslymonitor an area of interest included in one or more images captured bythe image sensor. For example, a user may be occupied by a certain task,for example, typing on a laptop, while image sensor 220 remains orientedin a particular direction and continuously monitors a portion of eachimage from a series of images to detect a trigger or other event. Forexample, image sensor 210 may be oriented towards a piece of laboratoryequipment and monitoring module 603 may be configured to monitor astatus light on the laboratory equipment for a change in status, whilethe user's attention is otherwise occupied.

In some embodiments consistent with the present disclosure, capturingunit 710 may include a plurality of image sensors 220. The plurality ofimage sensors 220 may each be configured to capture different imagedata. For example, when a plurality of image sensors 220 are provided,the image sensors 220 may capture images having different resolutions,may capture wider or narrower fields of view, and may have differentlevels of magnification. Image sensors 220 may be provided with varyinglenses to permit these different configurations. In some embodiments, aplurality of image sensors 220 may include image sensors 220 havingdifferent orientations. Thus, each of the plurality of image sensors 220may be pointed in a different direction to capture different images. Thefields of view of image sensors 220 may be overlapping in someembodiments. The plurality of image sensors 220 may each be configuredfor orientation adjustment, for example, by being paired with an imageadjustment unit 705. In some embodiments, monitoring module 603, oranother module associated with memory 550, may be configured toindividually adjust the orientations of the plurality of image sensors220 as well as to turn each of the plurality of image sensors 220 on oroff as may be required. In some embodiments, monitoring an object orperson captured by an image sensor 220 may include tracking movement ofthe object across the fields of view of the plurality of image sensors220.

Embodiments consistent with the present disclosure may includeconnectors configured to connect a capturing unit and a power unit of awearable apparatus. Capturing units consistent with the presentdisclosure may include at least one image sensor configured to captureimages of an environment of a user. Power units consistent with thepresent disclosure may be configured to house a power source and/or atleast one processing device. Connectors consistent with the presentdisclosure may be configured to connect the capturing unit and the powerunit, and may be configured to secure the apparatus to an article ofclothing such that the capturing unit is positioned over an outersurface of the article of clothing and the power unit is positionedunder an inner surface of the article of clothing. Exemplary embodimentsof capturing units, connectors, and power units consistent with thedisclosure are discussed in further detail with respect to FIGS. 8-14.

FIG. 8 is a schematic illustration of an embodiment of wearableapparatus 110 securable to an article of clothing consistent with thepresent disclosure. As illustrated in FIG. 8, capturing unit 710 andpower unit 720 may be connected by a connector 730 such that capturingunit 710 is positioned on one side of an article of clothing 750 andpower unit 720 is positioned on the opposite side of the clothing 750.In some embodiments, capturing unit 710 may be positioned over an outersurface of the article of clothing 750 and power unit 720 may be locatedunder an inner surface of the article of clothing 750. The power unit720 may be configured to be placed against the skin of a user.

Capturing unit 710 may include an image sensor 220 and an orientationadjustment unit 705 (as illustrated in FIG. 7). Power unit 720 mayinclude mobile power source 520 and processor 210. Power unit 720 mayfurther include any combination of elements previously discussed thatmay be a part of wearable apparatus 110, including, but not limited to,wireless transceiver 530, feedback outputting unit 230, memory 550, anddata port 570.

Connector 730 may include a clip 715 or other mechanical connectiondesigned to clip or attach capturing unit 710 and power unit 720 to anarticle of clothing 750 as illustrated in FIG. 8. As illustrated, clip715 may connect to each of capturing unit 710 and power unit 720 at aperimeter thereof, and may wrap around an edge of the article ofclothing 750 to affix the capturing unit 710 and power unit 720 inplace. Connector 730 may further include a power cable 760 and a datacable 770. Power cable 760 may be capable of conveying power from mobilepower source 520 to image sensor 220 of capturing unit 710. Power cable760 may also be configured to provide power to any other elements ofcapturing unit 710, e.g., orientation adjustment unit 705. Data cable770 may be capable of conveying captured image data from image sensor220 in capturing unit 710 to processor 800 in the power unit 720. Datacable 770 may be further capable of conveying additional data betweencapturing unit 710 and processor 800, e.g., control instructions fororientation adjustment unit 705.

FIG. 9 is a schematic illustration of a user 100 wearing a wearableapparatus 110 consistent with an embodiment of the present disclosure.As illustrated in FIG. 9, capturing unit 710 is located on an exteriorsurface of the clothing 750 of user 100. Capturing unit 710 is connectedto power unit 720 (not seen in this illustration) via connector 730,which wraps around an edge of clothing 750.

In some embodiments, connector 730 may include a flexible printedcircuit board (PCB). FIG. 10 illustrates an exemplary embodiment whereinconnector 730 includes a flexible printed circuit board 765. Flexibleprinted circuit board 765 may include data connections and powerconnections between capturing unit 710 and power unit 720. Thus, in someembodiments, flexible printed circuit board 765 may serve to replacepower cable 760 and data cable 770. In alternative embodiments, flexibleprinted circuit board 765 may be included in addition to at least one ofpower cable 760 and data cable 770. In various embodiments discussedherein, flexible printed circuit board 765 may be substituted for, orincluded in addition to, power cable 760 and data cable 770.

FIG. 11 is a schematic illustration of another embodiment of a wearableapparatus securable to an article of clothing consistent with thepresent disclosure. As illustrated in FIG. 11, connector 730 may becentrally located with respect to capturing unit 710 and power unit 720.Central location of connector 730 may facilitate affixing apparatus 110to clothing 750 through a hole in clothing 750 such as, for example, abutton-hole in an existing article of clothing 750 or a specialty holein an article of clothing 750 designed to accommodate wearable apparatus110.

FIG. 12 is a schematic illustration of still another embodiment ofwearable apparatus 110 securable to an article of clothing. Asillustrated in FIG. 12, connector 730 may include a first magnet 731 anda second magnet 732. First magnet 731 and second magnet 732 may securecapturing unit 710 to power unit 720 with the article of clothingpositioned between first magnet 731 and second magnet 732. Inembodiments including first magnet 731 and second magnet 732, powercable 760 and data cable 770 may also be included. In these embodiments,power cable 760 and data cable 770 may be of any length, and may providea flexible power and data connection between capturing unit 710 andpower unit 720. Embodiments including first magnet 731 and second magnet732 may further include a flexible PCB 765 connection in addition to orinstead of power cable 760 and/or data cable 770. In some embodiments,first magnet 731 or second magnet 732 may be replaced by an objectcomprising a metal material.

FIG. 13 is a schematic illustration of yet another embodiment of awearable apparatus 110 securable to an article of clothing. FIG. 13illustrates an embodiment wherein power and data may be wirelesslytransferred between capturing unit 710 and power unit 720. Asillustrated in FIG. 13, first magnet 731 and second magnet 732 may beprovided as connector 730 to secure capturing unit 710 and power unit720 to an article of clothing 750. Power and/or data may be transferredbetween capturing unit 710 and power unit 720 via any suitable wirelesstechnology, for example, magnetic and/or capacitive coupling, near fieldcommunication technologies, radiofrequency transfer, and any otherwireless technology suitable for transferring data and/or power acrossshort distances.

FIG. 14 illustrates still another embodiment of wearable apparatus 110securable to an article of clothing 750 of a user. As illustrated inFIG. 14, connector 730 may include features designed for a contact fit.For example, capturing unit 710 may include a ring 733 with a hollowcenter having a diameter slightly larger than a disk-shaped protrusion734 located on power unit 720. When pressed together with fabric of anarticle of clothing 750 between them, disk-shaped protrusion 734 may fittightly inside ring 733, securing capturing unit 710 to power unit 720.FIG. 14 illustrates an embodiment that does not include any cabling orother physical connection between capturing unit 710 and power unit 720.In this embodiment, capturing unit 710 and power unit 720 may transferpower and data wirelessly. In alternative embodiments, capturing unit710 and power unit 720 may transfer power and data via at least one ofcable 760, data cable 770, and flexible printed circuit board 765.

FIG. 15 illustrates another aspect of power unit 720 consistent withembodiments described herein. Power unit 720 may be configured to bepositioned directly against the user's skin. To facilitate suchpositioning, power unit 720 may further include at least one surfacecoated with a biocompatible material 740. Biocompatible materials 740may include materials that will not negatively react with the skin ofthe user when worn against the skin for extended periods of time. Suchmaterials may include, for example, silicone, PTFE, kapton, polyimide,titanium, nitinol, platinum, and others. Also as illustrated in FIG. 15,power unit 720 may be sized such that an inner volume of the power unitis substantially filled by mobile power source 520. That is, in someembodiments, the inner volume of power unit 720 may be such that thevolume does not accommodate any additional components except for mobilepower source 520. In some embodiments, mobile power source 520 may takeadvantage of its close proximity to the skin of user's skin. Forexample, mobile power source 520 may use the Peltier effect to producepower and/or charge the power source.

In further embodiments, an apparatus securable to an article of clothingmay further include protective circuitry associated with power source520 housed in power unit 720. FIG. 16 illustrates an exemplaryembodiment including protective circuitry 775. As illustrated in FIG.16, protective circuitry 775 may be located remotely with respect topower unit 720. In alternative embodiments, protective circuitry 775 mayalso be located in capturing unit 710, on flexible printed circuit board765, or in power unit 720.

Protective circuitry 775 may be configured to protect image sensor 220and/or other elements of capturing unit 710 from potentially dangerouscurrents and/or voltages produced by mobile power source 520. Protectivecircuitry 775 may include passive components such as capacitors,resistors, diodes, inductors, etc., to provide protection to elements ofcapturing unit 710. In some embodiments, protective circuitry 775 mayalso include active components, such as transistors, to provideprotection to elements of capturing unit 710. For example, in someembodiments, protective circuitry 775 may comprise one or more resistorsserving as fuses. Each fuse may comprise a wire or strip that melts(thereby braking a connection between circuitry of image capturing unit710 and circuitry of power unit 720) when current flowing through thefuse exceeds a predetermined limit (e.g., 500 milliamps, 900 milliamps,1 amp, 1.1 amps, 2 amp, 2.1 amps, 3 amps, etc.) Any or all of thepreviously described embodiments may incorporate protective circuitry775.

In some embodiments, the wearable apparatus may transmit data to acomputing device (e.g., a smartphone, tablet, watch, computer, etc.)over one or more networks via any known wireless standard (e.g.,cellular, Wi-Fi, Bluetooth®, etc.), or via near-filed capacitivecoupling, other short range wireless techniques, or via a wiredconnection. Similarly, the wearable apparatus may receive data from thecomputing device over one or more networks via any known wirelessstandard (e.g., cellular, Wi-Fi, Bluetooth®, etc.), or via near-fieldcapacitive coupling, other short range wireless techniques, or via awired connection. The data transmitted to the wearable apparatus and/orreceived by the wireless apparatus may include images, portions ofimages, identifiers related to information appearing in analyzed imagesor associated with analyzed audio, or any other data representing imageand/or audio data. For example, an image may be analyzed and anidentifier related to an activity occurring in the image may betransmitted to the computing device (e.g., the “paired device”). In theembodiments described herein, the wearable apparatus may process imagesand/or audio locally (on board the wearable apparatus) and/or remotely(via a computing device). Further, in the embodiments described herein,the wearable apparatus may transmit data related to the analysis ofimages and/or audio to a computing device for further analysis, display,and/or transmission to another device (e.g., a paired device). Further,a paired device may execute one or more applications (apps) to process,display, and/or analyze data (e.g., identifiers, text, images, audio,etc.) received from the wearable apparatus.

Some of the disclosed embodiments may involve systems, devices, methods,and software products for determining at least one keyword. For example,at least one keyword may be determined based on data collected byapparatus 110. At least one search query may be determined based on theat least one keyword. The at least one search query may be transmittedto a search engine.

In some embodiments, at least one keyword may be determined based on atleast one or more images captured by image sensor 220. In some cases,the at least one keyword may be selected from a keywords pool stored inmemory. In some cases, optical character recognition (OCR) may beperformed on at least one image captured by image sensor 220, and the atleast one keyword may be determined based on the OCR result. In somecases, at least one image captured by image sensor 220 may be analyzedto recognize: a person, an object, a location, a scene, and so forth.Further, the at least one keyword may be determined based on therecognized person, object, location, scene, etc. For example, the atleast one keyword may comprise: a person's name, an object's name, aplace's name, a date, a sport team's name, a movie's name, a book'sname, and so forth.

In some embodiments, at least one keyword may be determined based on theuser's behavior. The user's behavior may be determined based on ananalysis of the one or more images captured by image sensor 220. In someembodiments, at least one keyword may be determined based on activitiesof a user and/or other person. The one or more images captured by imagesensor 220 may be analyzed to identify the activities of the user and/orthe other person who appears in one or more images captured by imagesensor 220. In some embodiments, at least one keyword may be determinedbased on at least one or more audio segments captured by apparatus 110.In some embodiments, at least one keyword may be determined based on atleast GPS information associated with the user. In some embodiments, atleast one keyword may be determined based on at least the current timeand/or date.

In some embodiments, at least one search query may be determined basedon at least one keyword. In some cases, the at least one search querymay comprise the at least one keyword. In some cases, the at least onesearch query may comprise the at least one keyword and additionalkeywords provided by the user. In some cases, the at least one searchquery may comprise the at least one keyword and one or more images, suchas images captured by image sensor 220. In some cases, the at least onesearch query may comprise the at least one keyword and one or more audiosegments, such as audio segments captured by apparatus 110.

In some embodiments, the at least one search query may be transmitted toa search engine. In some embodiments, search results provided by thesearch engine in response to the at least one search query may beprovided to the user. In some embodiments, the at least one search querymay be used to access a database.

For example, in one embodiment, the keywords may include a name of atype of food, such as quinoa, or a brand name of a food product; and thesearch will output information related to desirable quantities ofconsumption, facts about the nutritional profile, and so forth. Inanother example, in one embodiment, the keywords may include a name of arestaurant, and the search will output information related to therestaurant, such as a menu, opening hours, reviews, and so forth. Thename of the restaurant may be obtained using OCR on an image of signage,using GPS information, and so forth. In another example, in oneembodiment, the keywords may include a name of a person, and the searchwill provide information from a social network profile of the person.The name of the person may be obtained using OCR on an image of a nametag attached to the person's shirt, using face recognition algorithms,and so forth. In another example, in one embodiment, the keywords mayinclude a name of a book, and the search will output information relatedto the book, such as reviews, sales statistics, information regardingthe author of the book, and so forth. In another example, in oneembodiment, the keywords may include a name of a movie, and the searchwill output information related to the movie, such as reviews, boxoffice statistics, information regarding the cast of the movie, showtimes, and so forth. In another example, in one embodiment, the keywordsmay include a name of a sport team, and the search will outputinformation related to the sport team, such as statistics, latestresults, future schedule, information regarding the players of the sportteam, and so forth. For example, the name of the sport team may beobtained using audio recognition algorithms.

Generating and Displaying Face Clouds

As used herein, a “face cloud” refers to any arrangement of a pluralityof visual representations of people's faces, where the arrangement isdetermined, at least in part, by one or more parameters and notrandomly. For example, as explained below, linking attributes may beused to arrange the plurality of visual representations in a face cloud.

In some embodiments, a wearable apparatus of the present disclosure maygenerate a customized user interface comprising a face cloud. The facecloud may be displayed over a display device comprised in or associatedwith the wearable apparatus. The face cloud may improve the userexperience by providing a visual interface that summarizes a user'sinteractions. Accordingly, embodiments of the present disclosure providefor improved display interfaces, particularly for wearable devices.

Some embodiments of the present disclosure, rather than usingconventional user interface methods to display a generic list of people,instead use a specific manner of displaying a limited set of informationto the user in a visual and easily perceptible manner. Extant interfacesfor summarizing a user's interactions are heavily text-based anddifficult to manipulate. Accordingly, embodiments of the presentdisclosure provide for improved display interfaces that are organizedvisually and allow for easy manipulation by the user.

FIG. 17 illustrates an exemplary embodiment of a memory containingsoftware modules consistent with the present disclosure. Included inmemory 1700 are person identification module 1701, affinity analysismodule 1702, image representation module 1703, and visualization module1704. Modules 1701, 1702, 1703, and 1704 may contain softwareinstructions for execution by at least one processing device, e.g.,processor 210, included with a wearable apparatus. In some embodiments,memory 1700 may comprise a non-transitory computer readable medium thatresides on a mobile communications device (e.g., a smart phone)configured to be wirelessly paired with wearable image sensor 220 ofcapturing unit 710.

Person identification module 1701, affinity analysis module 1702, imagerepresentation module 1703, and visualization module 1704 may cooperateto generate and present face clouds to a user of wireless apparatus 110,as well as cooperating to allow interaction therewith. Memory 1700 maybe separate from and/or integrated with memory 550 of FIG. 6, describedabove. Similarly, orientation identification module 601, orientationadjustment module 602, and monitoring module 603 of FIG. 6 may operatein tandem or concurrently with person identification module 1701,affinity analysis module 1702, image representation module 1703, andvisualization module 1704 of FIG. 17.

Person identification module 1701 may be configured to receive aplurality of images from an environment of a user of wearable apparatus110 captured by image sensor 220 of capturing unit 710 and analyze theplurality of images to identify a plurality of people. As discussedherein, received images or facial images may vary in size, shape, andcontent. For example, in some embodiments, an image including a widerscene (e.g., a plurality of individuals, or an individual and his or hersurrounding environment) may be cropped or otherwise truncated to aface, a portion of a face, a headshot of a person (e.g., head andshoulders), or a portion of a person's body including the person's face.In other embodiments, received images may include such wider scenes(e.g., a plurality of individuals, or an individual and his or hersurrounding environment). In still other embodiments, received imagesmay include a face, a portion of a face, a headshot of a person (e.g.,head and shoulders), or a portion of a person's body including theperson's face.

Person identification module 1701 may identify persons in receivedimages using any known techniques. For example, person identificationmodule 1701 may perform image processing on a currently received facialimage and determine that the received facial image corresponds to a faceof a recognized individual whose facial image has been previously storedin memory. Thus, person identification module 1701 may performpixel-by-pixel matching. In other embodiments, person identificationmodule 1701 may perform more sophisticated techniques. For example,person identification module 1701 may use one or more classifiers (orcascading classifiers) to derive an identity of a person included in theplurality of images. Additionally or alternatively, personidentification module 1701 may use a neural network to identify one ormore likely identifications of a person included in the plurality ofimages. Alternatively, person identification module 1701 may use aneural network to derive a plurality of features (e.g., a feature set)from one or more of the images and then map the derived features (orfeature set) onto one or more likely identifications. The mapping may beperformed using simple correlation or using more sophisticatedtechniques, such as another neural network.

In any of the embodiments above, likely matches may have associatedprobabilities, e.g., output from the neural network; calculated based onpercentage of matching pixels, features, etc.; or the like. Accordingly,person identification module 1701 may select an identification of aperson from a plurality of likely matches by selecting the highestprobability match.

In any of the embodiments above, one or more of the plurality of imagesmay include more than one person or no persons. Person identificationmodule 1701 may therefore use a bounding box architecture, such as YouOnly Look Once (YOLO) or Single-Shot Detector (SSD), to identifybounding boxes including a person therein. For example, if personidentification module 1701 uses a bounding box architecture and receivesno bounding boxes classified as persons or receives bounding boxes allhaving person classifications below a threshold (e.g., less than 50%,less than 40%, less than 25%, less than 10%, less than 5%, or the like),person identification module 1701 may discard the image out of theplurality of images as including no persons. Similarly, if personidentification module 1701 uses a bounding box architecture and receivesa plurality of bounding boxes classified as persons or receives aplurality of bounding boxes having person classifications above athreshold (e.g., more than 50%, more than 40%, more than 25%, more than10%, more than 5%, or the like), person identification module 1701 mayperform identification on portions of the image defined by the pluralityof bounding boxes.

In any of the embodiments above, person identification module 1701 mayreplace for use in a face cloud or other visualization (e.g., generatedby visualization module 1704), a previously stored facial image with acurrently received facial image. Accordingly, person identificationmodule 1701 may ensure that stored facial images are up-to-date. In suchembodiments, person identification module 1701 may update the storedfacial image only if the one or more quality measures of the currentlyreceived facial image exceed one or more quality measures of thepreviously stored facial image. For example, in some embodiments, personidentification module 1701 may update the stored facial image only if aresolution, a sharpness, a peak signal to noise ratio (PSNR), astructural similarity (SSIM), a visual information fidelity (VIF), orthe like of the currently received facial image are higher than the same(or a similar) measure for the previously stored facial image. Inanother example, person identification module 1701 may update the storedfacial image only if a noise or the like of the currently receivedfacial image are lower than the same (or a similar) measure for thepreviously stored facial image.

In any of the embodiments above, person identification module 1701 mayreceive a plurality of facial images including a plurality of facialimages of a same individual. In such embodiments, person identificationmodule 1701 may perform image processing on the plurality of images ofthe same individual to determine an existence of a plurality of imagesof the same individual, and select a single image of the individual foruse (e.g., for presentation in a face cloud determined by visualizationmodule 1704).

Affinity analysis module 1702 may be configured to analyze the pluralityof images to determine an affinity level between the user and each ofthe plurality of people. For example, affinity analysis module 1702 mayanalyze the plurality of images received by person identification module1701. Affinity analysis module 1702 may calculate the affinity levelbased on a relationship between the user of wearable apparatus 110 andindividuals identified by person identification module 1701. Forexample, affinity analysis module 1702 may determine whether the userand the identified person are family (e.g., siblings, parents,grandparents, cousins, or the like), friends, coworkers, or the like. Insuch embodiments, affinity analysis module 1702 may use information fromone or more social media accounts of the user (e.g., Facebook,Instagram, Twitter, or the like) to assist with determining therelationship.

Additionally or alternatively, affinity analysis module 1702 maydetermine the affinity level in accordance with a time or frequency atwhich the facial image was captured. Thus, affinity analysis module 1702may determine the affinity level based on a number of interactionsbetween the user and the identified person. For example, affinityanalysis module 1702 may determine how many times the user had aninteraction with the identified person during the last day, the lastweek, the last month, or the like. Accordingly, affinity analysis module1702 may use historical information about the user (e.g., previousimages, videos, audio, or the like) to determine the number ofinteractions. Additionally or alternatively, affinity analysis module1702 may use a time of capture and/or a place of capture of images inwhich the identified person appears to determine an affinity level. Forexample, a person appearing in an image captured at home may bedetermined to have a higher affinity level than a person appearing in animage captured at a bar or a club. In another example, a personappearing in an image captured during dinner may be determined to have ahigher affinity level than a person appearing in an image capturedduring a work day. Additionally or alternatively, the user may indicatelevels of affinity with identified persons, e.g., by grouping knownpersons into clusters such as “friends,” “acquaintances,” or the like.

In some embodiments, affinity analysis module 1702 may operate withoutperson identification module 1701. For example, affinity analysis module1702 may receive, from wearable image sensor 220 of capturing unit 710,a plurality of facial images of individuals with whom a user of wearableapparatus 110 interacted and may store, in association with each of theplurality of facial images, at least one linking attribute. In such anexample, the at least one linking attribute may include at least one ofan environmental characteristic, a geographical location, an associatedword, a social interconnection, or a temporal indicator.

The environmental characteristic may include at least one of abackground characteristic and an object in a vicinity of a facial image.For example, the environmental characteristic may include a brightnessof the background such that the linking attribute may include ‘inside’or ‘outside.’ In another example, the environmental characteristic mayinclude an object such as a coffee cup such that the linking attributeincludes ‘coffee.’ An example of the latter is depicted in FIG. 18F,described below.

The geographical location may include at least one of an event, aphysical space, and a GPS coordinate. For example, the geographicallocation may include a ‘work retreat’ or ‘concert.’ In another example,the geographical location may include a ‘coffee shop’ or a ‘neighbor'shouse.’ An example of this is depicted in FIG. 18E, described below.

An associated word may include at least one of a word derived from asound file associated with a facial image and a word previouslyassociated with an individual of the facial image. A microphone ofwearable apparatus 110 may capture the sound file and time stamp of thesound file such that it may be synchronized with time stamps of imagesreceived from wearable apparatus 110. In one example, affinity analysismodule 1702 may receive a list of words parsed from a sound filecaptured during a time period associated with detection of a presence ofthe plurality of individuals in a vicinity of the wearer and associatein memory, the list of parsed words with a corresponding individual. Thewords may be parsed using any audio analysis technique, such as limitedvocabulary recognition, large vocabulary recognition, with or withoutspeaker separation, recognition, verification, or the like. In someembodiments, the voice analysis operations may be enhanced in accordancewith training one or more engines upon the user's voice and optionallythe voices of people close to the user, such as the user's spouse.Accordingly, affinity analysis module 1702 may transcribe the sound filein order to parse words therefrom and then store the parsed words inassociation with corresponding facial images.

Additionally or alternatively, affinity analysis module 1702 mayassociate in memory a frequency of use indication associated with eachparsed word. For example, the frequency of use indication may include ahistogram associated with each word in a word list. Accordingly, theword list may include key words extracted from a conversation. In suchembodiments, affinity analysis module 1702 may eliminate common wordssuch as articles (‘the,’ ‘a,’ ‘an,’ or the like), prepositions (‘of,’‘from,’ ‘to; or the like), or the like from the histogram (or beforedetermining the histogram).

The social interconnection may include at least one of an individualdetected proximate a field of view of a facial image and otherwiserecognized individuals. For example, an individual detected proximate toother recognized individuals (e.g., matching one or more knownindividuals in memory) may be determined as having a socialinterconnection with the other recognized individuals. Additionally oralternatively, as explained above, affinity analysis module 1702 may usesocial media information to determine social interconnections. Forexample, individuals may be determined to have social interconnectionsbased on the level of connection between a recognized individual and theuser.

The temporal indicator may include at least one of a date and a timestamp associated with a facial image. For example, facial images havingtime stamps and/or date stamps that are the same (or are in proximity,e.g., within 15 minutes, within an hour, within a day, within a week, orthe like) may be linked together.

These embodiments may be combined with corresponding identificationsfrom person identification module 1701. For example, affinity analysismodule 1702 may store, in association with each of the plurality offacial images identified by person identification module 1701, the atleast one linking attribute. Affinity analysis module 1702 may alsostore the corresponding identification from person identification module1701 in association with each of the plurality of facial images.

In any of the embodiments above, linking attributes may be associatedwith each individual in a histographic manner. For example, eachindividual (e.g., identified by person identification module 1701) mayhave an associated histogram including one or more linking attributes.The histogram may indicate the level of connection between theindividual and the user (and/or other individuals) based on each linkingattribute. Additionally or alternatively, affinity analysis module 1702may use the histogram to determine the level of connection between theindividual and the user (and/or other individuals) based on each linkingattribute in the histogram. For example, affinity analysis module 1702may use one or more word frequencies included in the histogram to linkindividuals having histograms with the same words (or similar words,such as synonyms) having the same or similar (e.g., within 1 word perday, within 5 words per day, or the like) frequencies.

Image representation module 1703 may be configured to obtain an imagerepresentation of each of the plurality of people. For example, imagerepresentation module 1703 may obtain image representations from amemory storing a plurality of image representations indexed byidentification.

Accordingly, image representation module 1703 may obtain images based onidentifications from person identification module 1701.

In embodiments where affinity analysis module 1702 calculates or obtainsa list of words parsed from a sound file, image representation module1703 may select a specific parsed word and choose the subset of facialimages for display based on the selection. For example, imagerepresentation module 1703 may select the specific parsed word, obtain alist of individuals based on stored associations between individuals andparsed words, and obtain image representations of the list ofindividuals. In embodiments where each individual has an associatedhistogram, image representation module 1703 may select the specificparsed word, obtain a list of individuals based on stored histogramsassociated with individuals, and obtain image representations of thelist of individuals.

In some embodiments, the selecting may occur as a result of input fromthe user, based on context in a current environment of the user, basedon words used in a current conversation, or based on an object in arecent field of view of the wearable image sensor. For example, the usermay input the selected word that image representation module 1703 uses,to obtain a list of individuals and corresponding image representations.In another example, a context in an environment of the user of wearableapparatus 110 may include a location (such as work, a store, a coffeeshop, or the like), one or more recognized individuals, one or morewords (e.g., printed on a sign in the environment), or the like.Accordingly, a description of the location, a relationship between therecognized individual(s) and the user, a word in the environment, or thelike, may be selected as the selected word. In yet another example, aword used in a current conversation (e.g., recorded by a microphone ofwearable apparatus 110) may be selected as the selected word. Imagerepresentation module 1703 may be configured to select a word mostfrequently used in a current conversation but may also eliminate commonwords such as articles, prepositions, or the like and/or proper names.In another example, an object identified by wearable apparatus 110, suchas a ‘coffee,’ a ‘cake,’ a ‘file,’ or the like, may be selected as theselected word.

Visualization module 1704 may be configured to generate, based on theaffinity levels (e.g., determined by affinity analysis module 1702), avisualization comprising the image representations (e.g., obtained byimage representation module 1703). Additionally or alternatively,visualization module 1704 may cause a visual representation to bedisplayed, the visual representation including the plurality of facialimages, and the plurality of facial images being arranged based on theat least one linking attribute. In some embodiments, the visualrepresentation may comprise a face cloud such that visualization module1704 is configured to display, in a face cloud, the plurality of facialimages, the plurality of facial images being arranged based on the atleast one linking attribute.

In embodiments where linking attributes are associated with eachindividual in a histographic manner, during display, the facial imagesmay be arranged based on histographic strength. For example, facialimages with stronger historgraphic strength may be arranged in proximityto each other, closer to a focal point, or the like. Additionally oralternatively, during display, facial images of individuals with greaterhistographic strength may be displayed larger, more prominently, and/orwith a different shape than facial images of individuals with lesserhistographic strength.

Additionally or alternatively, a prominence level in the visualizationof at least one of the image representations of a person from among theplurality of people may depend on the affinity level associated with theperson. For example, the prominence level may be indicated in thevisualization by at least one of size, shape, and color.

Additionally or alternatively, sizes, shapes, and/or colors of facialimages may be indicative of time spent with an individual and/orindicative of histographic strength to a linking attribute that is afocal point of the face cloud. For example, visualization module 1704may increase a size of a facial image if an individual associated withthe facial image appears in more images captured by wearable apparatus110, has more interactions with a user of wearable apparatus 110, orotherwise has spent more time with the user of wearable apparatus 110than another individual associated with another facial image. Similarly,visualization module 1704 may select a different color or shape of onefacial image if an individual associated with the facial image appearsin more images captured by wearable apparatus 110, has more interactionswith a user of wearable apparatus 110, or otherwise has spent more timewith the user of wearable apparatus 110 than another individualassociated with another facial image. In another example, example,visualization module 1704 may increase a size of a facial image if ahistogram of an individual associated with the facial image has a higherfrequency or other measure of a linking attribute than another histogramof another individual associated with another facial image. Similarly,visualization module 1704 may select a different color or shape of onefacial image if a histogram of an individual associated with the facialimage has a higher frequency or other measure of a linking attributethan another histogram of another individual associated with anotherfacial image.

For example, the linking attributes of each of the facial images mayinclude an importance level. Accordingly, in embodiments where size isadjusted, the facial images may be displayed in different sizesaccording to the importance levels. For example, people the user meetsmore often and/or for longer periods of time may be determined to bemore important and thus be displayed as larger.

Additionally or alternatively, the linking attribute may includeunrecognized individuals. For example, facial images may be adjustedbased on a level of connection between associated individuals and one ormore unrecognized individuals. The level of connection may be determinedusing a plurality of measures, e.g., based on a proximity between theassociated individuals and the one or more unrecognized individuals inthe received images, based on being participants in one or more of thesame conversations, based on connections determined using social mediainformation, or any combination thereof.

Additionally or alternatively, the linking attribute may be generatedthrough image analysis to determine context. For example, as explainedabove, a location (such as work, a coffee shop, or the like), one ormore recognized individuals, one or more words (e.g., printed on a signin the environment), or the like such that the linking attributecomprises the location, a relationship between recognized individualsand the user, the one or more words in the environment, or the like.

Additionally or alternatively, the linking attribute may be determinedat least on part, through reference to a calendar of the user. Forexample, visualization module 1704 may select a location, one or morerecognized individuals, one or more words, or the like by mapping timestamps of the received images to one or more events stored in thecalendar.

Additionally or alternatively, the linking attribute may be userdefined. For example, as explained above, the user may select anenvironmental characteristic, a geographical location, an associatedword, a social interconnection, a temporal indicator, or any otherlinking attribute.

In any of the embodiments using a linking attribute, visualizationmodule 1704 may, for a given face cloud, display an indication of atleast one linking attribute serving as a basis for organization of theface cloud. An example of such an indication is depicted in FIGS. 18Cand 18D, described below. In such embodiments, greater commonality to alinking attribute may be indicated on the display through proximity to afocal location of the face cloud.

In any of the embodiments using a word list, as described above withrespect to affinity analysis module 1703, visualization module 1704 maypresent, on a screen of a display (e.g., of wearable apparatus 110 or ofa mobile communications device wirelessly paired with wearable apparatus110), a subset of the plurality of facial images of individuals whoseword lists contain overlapping common words.

Additionally or alternatively, the set of the plurality of facial imagesmay be displayed based on a stored set of rules. For example, the storedset of rules may include at least one of most recent (such that only themost recent plurality of facial images are displayed in the face cloud),most often (such that only facial images appearing in a highest numberof images captured by wearable device 110 are displayed in the facecloud), and longest duration (such that only facial images being inimages captured by wearable device 110 for a longest length of time aredisplayed in the face cloud).

In any of the embodiments including a face cloud, displaying of the facecloud may include displaying only a portion of the face cloud existingin a focal region. For example, visualization module 1704 may generate aface cloud by organizing and/or arranging facial images according to oneor more linking attributes and then select the portion of the face cloudfor display based on the focal point and a range extending from thefocal point. The range, for example, may comprise a circle, an ellipse,a square, a rhombus, a rectangle, a parallelogram, or any other shape,whether regular or irregular.

In such embodiment, the focal region may be dynamically selected basedon a detected variable. For example, the detected variable may includethe presence of the mobile communication device in a particulargeographical area such that the focal region includes images ofindividuals previously associated with the particular geographical area.Additionally or alternatively, the detected variable may include thepresence of the mobile communication device in a vicinity of a knownindividual such that the focal region includes images of individualspreviously associated with the known individual. Additionally oralternatively, the detected variable may include at least one wordderived from a sound file (e.g., recorded by a microphone of wearableapparatus 110) such that the focal region includes images of individualspreviously associated with the at least one word.

In any of the embodiments above, causing the visual representation to bedisplayed may include transmitting information for generating thevisualization to a mobile communication device wirelessly paired withthe wearable apparatus. For example, visualization module 1704 maytransmit the face cloud or any other generated visualization via acomputer network, such as Bluetooth®, a WiFi network, a cellular network(such as 3G, 4G, LTE, or the like), or any other radio frequency orother wireless network.

In any of the embodiments above, the face cloud may be displayed in twodimensions or displayed as a virtual 3D representation. For example, the3D representation may comprise a 3D image displayed on a two-dimensionalscreen (e.g., of wearable apparatus 110 or of a mobile communicationsdevice wirelessly paired with wearable apparatus 110) or may comprise aholographic projection, e.g., formed using one or more lasers.

In some embodiments, the form of the face cloud may include a series ofhomogeneously sized and shaped facial images arranged in a clusterformation. Additionally or alternatively, the form of the face cloud mayinclude an arrangement of the plurality of facial images in a clustersuch that at least one of the facial images has either a size, shape, orcolor indicator that differs from other facial images in the cluster, asexplained above. For example, the form of the face cloud may include anarrangement of the plurality of facial images in a cluster, a prominencelevel of a facial image from among the plurality of facial imagesdepending on an affinity level between the user and a person depicted inthe facial image. Additionally or alternatively, displaying the facecloud may include arranging the facial images in the face cloud suchthat facial images sharing at least one common linking attribute aredisplayed or oriented adjacent to each other.

Visualization module 1704 may also be configured to enable interactionwith a generated face cloud or other visualization. For example,visualization module 1704 may be configured to enable a user to select aparticular facial image to be a focal point of the face cloud. In suchan example, visualization module 1704 may allow a user to click, tap, orotherwise select the particular facial image. In such embodiments,visualization module 1704 may receive a designation of a focal pointindividual such that the input causes a rearrangement of the face cloudwhile maintaining the focal point individual.

Additionally or alternatively, visualization module 1704 may beconfigured such that, upon touching a facial image displayed on a touchscreen, additional information appears. For example, the additionalinformation may include at least one of name, contact information, andlinking attribute information for the selected facial image. In someembodiments, a scope of information presented may depend on duration ofpressing and/or a level of pressure applied. For example, visualizationmodule 1704 may display a name associated with the touched facial imageif the press is short (e.g., less than 0.5 second) and/or light (e.g.,less than 50 pascals) but may display a name, contact information, andthe linking attribute if the press is long (e.g., more than 0.5 second)and/or hard (e.g., more than 50 pascals).

Additionally or alternatively, visualization module 1704 may enable auser to select at least one linking attribute, the arrangement of facesin the cloud occurring based on the selected at least one linkingattribute. For example, visualization module 1704 may generate aseparate interface with a list of linking attributes such that the usermay click, tap, or otherwise select a desired linking attribute. Inanother example, visualization module 1704 may generate a list oflinking attributes adjacent to a generated face cloud or othervisualization such that the user may click, tap, or otherwise select adesired linking attribute.

Accordingly, input may occur through selection of an attribute from apick list. Alternatively, input may occur through sound processing on avoice capture, e.g., captured via a microphone of wearable apparatus110.

In any of the embodiments above, the face cloud may be initiallyarranged based on at least a first common attribute, and in response tothe input, the face cloud may be rearranged based on a second commonattribute. An example of such rearrangement is depicted in FIG. 18H,described below.

In any of the embodiments above, the face cloud may be displayed inmotion. For example, the face cloud may be displayed as a flow and maymove or change as images come and go, such as on a street where peoplemove in various directions. Additionally or alternatively, the facecloud may dynamically change based on a location of the user of wearabledevice 110 and/or people visible in a field of view of image sensor 220of capturing unit 710.

In embodiments where modules 1701, 1702, 1703, and 1704 run on a mobilecommunications device, visualization module 1704 may be configured torearrange the face cloud in response to an input received via the mobilecommunications device. For example, visualization module 1704 mayreceive input including a signal received from an accelerometer in themobile communications device that is reflective of a shaking of themobile communications device and rearrange the face cloud accordingly.

FIG. 18A illustrates an example of a face cloud having prominence levelsindicated by size consistent with the present disclosure. For example,as depicted in FIG. 18A, a plurality of facial images (e.g., images 1801a, 1801 b, 1801 c, 1801 d, 1801 e, and 1801 f) are displayed in acluster. Although depicted as within shapes (circles in the example ofFIG. 18A), the facial images may be displayed without shapes.

In the example of FIG. 18A, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on prominence levels (e.g., based on affinity levels),importance levels, histographic strength of a linking attribute, or thelike, as described above with respect to visualization module 1704 ofmemory 1700. Although not depicted in FIG. 18A, distances between thefacial images and a focal point of the face cloud or between facialimages and/or thickness of connecting lines (not pictured) may varybased on prominence levels (e.g., based on affinity levels), importancelevels, histographic strength to a linking attribute, or the like, asdescribed above with respect to visualization module 1704 of memory1700.

FIG. 18B illustrates an example of a face cloud having prominence levelsindicated by shape consistent with the present disclosure. For example,as depicted in FIG. 18B, a plurality of facial images (e.g., images 1803a, 1803 b, 1803 c, 1803 d, 1803 e, and 1803 f) are displayed in acluster. Moreover, in the example of FIG. 18B, the shapes enclosing ofthe facial images vary. For example, the shapes may vary based onprominence levels (e.g., based on affinity levels), importance levels,histographic strength to a linking attribute, or the like, as describedabove with respect to visualization module 1704 of memory 1700. Althoughnot depicted in FIG. 18B, distances between the facial images and afocal point of the face cloud or between facial images and/or thicknessof connecting lines (not pictured) may vary based on prominence levels(e.g., based on affinity levels), importance levels, histographicstrength to a linking attribute, or the like, as described above withrespect to visualization module 1704 of memory 1700.

FIG. 18C illustrates an example of a face cloud including a linking wordconsistent with the present disclosure. For example, as depicted in FIG.18C, a plurality of facial images (e.g., images 1805 a, 1805 b, 1805 c,1805 d, 1805 e, and 1805 f) are clustered around an indication 1807 of alinking word (e.g., “Birthday”). Although depicted as within shapes(circles in the example of FIG. 18C), the facial images may be displayedwithout shapes.

In the example of FIG. 18C, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on histographic strength to a linking attribute, or the like,as described above with respect to visualization module 1704 of memory1700. As further depicted in FIG. 18C, distances between the facialimages and the indication 1807 (or between facial images and/orthickness of connecting lines), vary based on histographic strength to alinking attribute, as described above with respect to visualizationmodule 1704 of memory 1700.

FIG. 18D illustrates an example of a face cloud including a plurality oflinking words (e.g., “Neighbors” and “Coworkers”) consistent with thepresent disclosure. For example, as depicted in FIG. 18D, a plurality offacial images (e.g., images 1809 a, 1809 b, 1809 c, 1809 d, 1809 e, and1809 f) are clustered around indications 1811 a and 1811 b of linkingwords. Although depicted as within shapes (circles in the example ofFIG. 18D), the facial images may be displayed without shapes.

In the example of FIG. 18D, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on histographic strength to a linking attribute, or the like,as described above with respect to visualization module 1704 of memory1700. As further depicted in FIG. 18D, distances between the facialimages and the indications 1811 a and 1811 b (or between facial imagesand/or thickness of connecting lines) vary based on histographicstrength to the corresponding linking attribute, as described above withrespect to visualization module 1704 of memory 1700.

FIG. 18E illustrates an example of a face cloud including a linkinggeographic location consistent with the present disclosure. For example,as depicted in FIG. 18E, a plurality of facial images (e.g., images 1813a, 1813 b, 1813 c, 1813 d, 1813 e, and 18130 are clustered around ageographical location 1815 (e.g., a room, a particular person's home,etc.). Although depicted as within shapes (circles in the example ofFIG. 18E), the facial images may be displayed without shapes.

In the example of FIG. 18E, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on histographic strength to a linking attribute, or the like,as described above with respect to visualization module 1704 of memory1700. As further depicted in FIG. 18E, distances between the facialimages and the geographical location 1815 (or between facial imagesand/or thickness of connecting lines) vary based on histographicstrength to the corresponding linking attribute, as described above withrespect to visualization module 1704 of memory 1700.

FIG. 18F illustrates an example of a face cloud including a linkingconcept or environmental characteristic consistent with the presentdisclosure. For example, as depicted in FIG. 18F, a plurality of facialimages (e.g., images 1817 a, 1817 b, 1817 c, 1817 d, 1817 e, and 1817 f)are clustered around an object 1819 (e.g., a coffee cup). Althoughdepicted as within shapes (circles in the example of FIG. 18F), thefacial images may be displayed without shapes.

In the example of FIG. 18F, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on histographic strength to a linking attribute, or the like,as described above with respect to visualization module 1704 of memory1700. As further depicted in FIG. 18F, distances between the facialimages and the object 1819 (or between facial images and/or thickness ofconnecting lines) vary based on histographic strength to thecorresponding linking attribute, as described above with respect tovisualization module 1704 of memory 1700.

FIG. 18G illustrates an example of a face cloud organized using atemporal indicator consistent with the present disclosure. For example,as depicted in FIG. 18G, a plurality of facial images (e.g., images 1821a, 1821 b, 1821 c, 1821 d, and 1821 e) are ordered with temporal (andcontext) indicators 1823 a, 1823 b, 1823 c, 1823 d, and 1823 e. Althoughdepicted as within shapes (circles in the example of FIG. 18G), thefacial images may be displayed without shapes.

Although depicted in FIG. 18G as having uniform sizes, the sizes of thefacial images (and the shapes enclosing the facial images) may vary insome embodiments. For example, the sizes may vary based on prominencelevels (e.g., based on affinity levels), importance levels, histographicstrength to a linking attribute, or the like, as described above withrespect to visualization module 1704 of memory 1700. Moreover, althoughnot depicted in FIG. 18G, distances between the facial images and atimeline of the face cloud may vary based on prominence levels (e.g.,based on affinity levels), importance levels, histographic strength to alinking attribute, or the like, as described above with respect tovisualization module 1704 of memory 1700.

FIG. 18H illustrates an example of designation of a focal pointindividual that causes a rearrangement of the face cloud consistent withthe present disclosure. As depicted in the example of FIG. 18H, aplurality of facial images (e.g., images 1825 a, 1825 b, 1825 c, 1825 d,1825 e, and 1825 f) are displayed in a cluster. Although depicted aswithin shapes (circles in the example of FIG. 18H), the facial imagesmay be displayed without shapes.

In the example of FIG. 18H, the sizes of the facial images (and theshapes enclosing the facial images) vary. For example, the sizes mayvary based on prominence levels (e.g., based on affinity levels),importance levels, histographic strength to a linking attribute, or thelike, as described above with respect to visualization module 1704 ofmemory 1700. Although not depicted in FIG. 18A, distances between thefacial images and a focal point of the face cloud may vary based onprominence levels (e.g., based on affinity levels), importance levels,histographic strength to a linking attribute, or the like, as describedabove with respect to visualization module 1704 of memory 1700.

As further depicted in FIG. 18H, facial image 1825 c may serve as adefault or initial focal individual in the face cloud. Accordingly, thesizes of other facial images and/or distances to facial image 1825 c maydepend on a level of interconnection between the individual associatedwith facial image 1825 c and other individuals associated with otherfacial images. Based on input from a user (e.g., via a click, a tap, orthe like), facial image 1825 e may be selected as the focal individual.Accordingly, the face cloud may be rearranged while maintaining thefocal point individual associated with facial image 1825 e. In theexample of FIG. 18H, sizes, placement, and distances of other facialimages may shift in accordance with the selected focal individual, e.g.,to show a level of interconnection between the individual associatedwith facial image 1825 e and other individuals associated with otherfacial images. Although not depicted in FIG. 18H, the selection may alsoresult in the removal of one or more facial images from the previousface cloud or the addition of one or more facial images to the new facecloud. Any of these changes may be animated such that the transformationdepicted in FIG. 18H is fluid rather than sudden.

Indeed, in any of the embodiments above, the generated face cloud may beanimated such that it comprises a movie or a video. For example, theface cloud may comprise a video such that at least one of the pluralityof facial images present in a first frame is not present in a second,later frame.

Additionally or alternatively, the face cloud may comprise a video suchthat at least one facial image not present in a first frame is presentin a second, later frame. In one implementation, the face cloud maycomprise a plurality of facial images of individuals present in alocation of the user and may comprise a video such that the plurality offacial images move within the face cloud as the user moves around in thelocation. Additionally or alternatively, the facial images may change asindividuals exit the location and enter the location or as the usermoves such that individuals are beyond a threshold distance of the userand that other individuals are within the threshold distance.

FIG. 19A is a flowchart of a method 1900 for generating a visualizationof image representations of a plurality of people. Method 1900 may beimplemented by a general-purpose computer or a special-purpose computerbuilt according to embodiments of the present disclosure. For example,method 1900 may be executed by at least one of processors 210, 210 a,210 b, and 540. In some embodiments, the steps of method 1900 may beperformed by wearable apparatus 110. In other embodiments, the steps ofmethod 1900 may be performed by wearable apparatus 110 and one or moreexternal devices (e.g., a processor included in an external server thatreceives data from wearable apparatus 110 over a network and/or aprocessor included in an external device such as a laptop, smartwatch,smartphone, tablet, ear phones, etc.).

At step 1901, the processor may analyze the plurality of images toidentify a plurality of people. For example, the processor may use anyof the analysis techniques described above with respect to personidentification module 1701 of FIG. 17.

At step 1903, the processor may analyze the plurality of images todetermine an affinity level between the user and each of the pluralityof people. For example, the processor may calculate the affinity levelas described above with respect to affinity analysis module 1702 of FIG.17.

At step 1905, the processor may obtain an image representation of eachof the plurality of people. For example, the processor may obtain theimage representations from a database, as explained above with respectto image representation module 1703 of FIG. 17.

At step 1907, the processor may generate, based on the affinity levels,a visualization comprising the image representations. For example, theprocessor may generate a face cloud, as explained above with respect tovisualization module 1704 of FIG. 17. Examples of generated face cloudsare also depicted in FIGS. 18A-18H, described above.

In some embodiments, method 1900 may include additional steps. Forexample, method 1900 may include transmitting the visualizationcomprising the image representations to a mobile communications devicewirelessly paired with the wearable apparatus. As explained above withrespect to visualization module 1704 of FIG. 17, the processor maytransmit the visualization via one or more computer networks and/orradio frequencies.

Moreover, any of the interactions described above with respect tovisualization module 1704 of FIG. 17 may be implemented in method 1900.For example, the generated visualization may be updated and/or animatedin response to input from the user.

FIG. 19B is a flowchart of a method 1910 for generating a visualrepresentation including a plurality of facial images arranged based onat least one linking attribute. Method 1910 of FIG. 19B may represent analternative to method 1900 for generating a face cloud. Method 1910 maybe implemented by a general-purpose computer or a special-purposecomputer built according to embodiments of the present disclosure. Forexample, method 1910 may be executed by at least one of processors 210,210 a, 210 b, and 540. hi some embodiments, the steps of method 1910 maybe performed by wearable apparatus 110. In other embodiments, the stepsof method 1910 may be performed by wearable apparatus 110 and one ormore external devices (e.g., a processor included in an external serverthat receives data from wearable apparatus 110 over a network and/or aprocessor included in an external device such as a laptop, smartwatch,smartphone, tablet, or ear phones, etc.).

At step 1911, the processor may receive from the wearable image sensor aplurality of facial images of individuals with whom the user interacted.For example, the processor may receive images from image sensor 220 ofcapturing unit 710.

In some embodiments, receiving a plurality of facial images may includereceiving a plurality of facial images of a same individual. In suchembodiments, the processor may further perform image processing on theplurality of images of the same individual, determine an existence of aplurality of images of the same individual, and select a single image ofthe individual for presentation in the face cloud of step 1915. Forexample, the processor may use any of the analysis techniques describedabove with respect to person identification module 1701 of FIG. 17.

In some embodiments, the processor may perform image processing on acurrently received facial image, and determine that the received facialimage corresponds to a face of a recognized individual with a facialimage previously stored in memory. In such embodiments, the processormay replace, for use in the face cloud of step 1915, the previouslystored facial image with the currently received facial image. Forexample, the processor may perform the replacement as described abovewith respect to person identification module 1701 of FIG. 17.

At step 1913, the processor may store, in association with each of theplurality of facial images, at least one linking attribute. The at leastone linking attribute may include at least one of an environmentalcharacteristic, a geographical location, an associated word, a socialinterconnection, or a temporal indicator. The at least one linkingattribute may be determined as described above with respect to affinityanalysis module 1702 of FIG. 17.

At step 1915, the processor may cause a visual representation to bedisplayed, the visual representation including the plurality of facialimages and/or may display, in a face cloud, the plurality of facialimages. The plurality of facial images may be arranged based on the atleast one linking attribute. For example, the processor may generate theface cloud as explained above with respect to visualization module 1704of FIG. 17. Examples of generated face clouds are also depicted in FIGS.18A-18H, described above.

In some embodiments, causing the visual representation to be displayedmay include transmitting information for generating the visualization toa mobile communications device wirelessly paired with the wearableapparatus. For example, as explained above with respect to visualizationmodule 1704 of FIG. 17, the processor may transmit the visualization viaone or more computer networks and/or radio frequencies.

In some embodiments, displaying may include arranging the facial imagesin the face cloud such that facial images sharing at least one commonlinking attribute are oriented or located adjacent to each other. Forexample, FIG. 18D is an example of facial images organized adjacent toother facial images sharing a common linking word.

In some embodiments, method 1910 may include additional steps. Forexample, method 1910 may include enabling a user to select a particularfacial image to be a focal point of the face cloud. Additionally oralternatively, method 1910 may include enabling a user to select atleast one linking attribute such that the arrangement of the face cloudoccurs based on the selected at least one linking attribute.Accordingly, any of the interactions described above with respect tovisualization module 1704 of FIG. 17 may be implemented in method 1910.

In some embodiments, method 1910 may be executed by a mobilecommunications device. In such embodiments, the processor may rearrangethe face cloud in response to an input received via the mobilecommunications device. Additionally or alternatively, the face cloud maybe initially arranged based on at least a first common attribute, and inresponse to the input, the face cloud may be rearranged based on asecond common attribute.

FIG. 19C is a flowchart of a method 1920 for generating a subset of aplurality of facial images of individuals having word lists withoverlapping common words. Method 1920 of FIG. 19C may represent analternative to method 1900 or method 1910 for generating a face cloud.Method 1920 may be implemented by a general-purpose computer or aspecial-purpose computer built according to embodiments of the presentdisclosure. For example, method 1920 may be executed by at least one ofprocessors 210, 210 a, 210 b, and 540. In some embodiments, the steps ofmethod 1920 may be performed by wearable apparatus 110. In otherembodiments, the steps of method 1920 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, ear phones,etc.).

At step 1921, the processor may receive from a wearable image sensorworn by a user a plurality of facial images of individuals with whom theuser has interacted. For example, the processor may receive images fromimage sensor 220 of capturing unit 710.

At step 1923, the processor may receive a list of words parsed from asound file captured during a time period associated with detection of apresence of the plurality of individuals in a vicinity of the wearer.For example, the processor may use any of the analysis techniquesdescribed above with respect to affinity analysis module 1702 of FIG.17.

At step 1925, the processor may associate in memory the list of parsedwords with a corresponding individual. hi some embodiments, theprocessor may further associate in memory a frequency of use indicationassociated with each parsed word. The parsed words may be determined asdescribed above with respect to affinity analysis module 1702 of FIG.17.

At step 1927, the processor may present on a screen of a display asubset of the plurality of facial images of individuals whose word listscontain overlapping common words. For example, the processor maygenerate a face cloud, as explained above with respect to visualizationmodule 1704 of FIG. 17. Examples of generated face clouds are alsodepicted in FIGS. 18A-18H, described above.

In some embodiments, method 1920 may include additional steps. Forexample, method 1920 may include selecting a specific parsed word andchoosing the subset of facial images for display based on the selection.In such embodiments, the selecting may occur as a result of input fromthe user, based on context in a current environment of the user, basedon words used in a current conversation, or based on an object in arecent field of view of the wearable image sensor. Accordingly, any ofthe interactions described above with respect to visualization module1704 of FIG. 17 may be implemented in method 1920.

Moreover, any of the generated visualizations or face clouds may beanimated. For example, selected facial images may appear to move closertoward a screen on which the visualization is shown as the identifiedperson moves closer to the user or the user moves closer to the person.In such an example, the facial image may disappear and/or appear to moveoff the screen as the identified person moves away from the user orpasses them while walking, or the user moves away or passes the person.Accordingly, the generated visualizations or face clouds may represent adynamic or real-time visualization of identified persons around theuser.

Indexing and Searching Persons and Objects

In some embodiments, a wearable apparatus of the present disclosure mayindex images of persons and objects in particular ways to assist withsearching. The database may associate facial images captured by awearable apparatus with key words from sound files, calendar dates,and/or contextual information from images captured by the wearableapparatus. Accordingly, by constructing and indexing the databaseaccording to the particular rules of the present disclosure, facialimages may be searched with greater speed and accuracy than inconventional systems.

In addition, rather than using conventional user interface methods todisplay search results, embodiments of the present disclosure insteadmay use a specific manner of displaying a limited set of information tothe user in a visual and easily perceptible manner. Conventionalinterfaces for displaying search results are heavily text-based anddifficult to manipulate. Accordingly, embodiments of the presentdisclosure provide for improved display interfaces that are organizedvisually and allow for easy manipulation by the user.

FIG. 20 illustrates an exemplary embodiment of a memory 2000 containingsoftware modules consistent with the present disclosure. Included inmemory 2000 are facial image module 2001, linking attribute module 2002,association module 2003, and display module 2004. Modules 2001, 2002,2003, and 2004 may contain software instructions for execution by atleast one processing device, e.g., processor 210, included with awearable apparatus. hi some embodiments, memory 2000 may comprise anon-transitory computer readable medium that resides on a mobilecommunications device (e.g., a smart phone) configured to be wirelesslypaired with wearable image sensor 220 of capturing unit 710.

Facial image module 2001, linking attribute module 2002, associationmodule 2003, and display module 2004 may cooperate to generate adatabase with indexed images and/or sounds from wireless apparatus 110,as well as cooperating to allow searches of the database and particulardisplays of results therefrom. Memory 2000 may be separate from and/orintegrated with memory 550 of FIG. 6, described above. Similarly,orientation identification module 601, orientation adjustment module602, and monitoring module 603 of FIG. 6 may operate in tandem orconcurrently with facial image module 2001, linking attribute module2002, association module 2003, and display module 2004 of FIG. 20.

Facial image module 2001 may be configured to receive images captured byimage sensor 220 of capturing unit 710. For example, the images may befrom an environment of a user of a wearable apparatus.

In some embodiments, the images may include an object image. The objectimage may include a facial image or be a facial image.

Additionally or alternatively, facial image module 2001 may receivefacial images of a plurality of individuals with whom a user of awearable apparatus interacted. For example, facial image module 2001 maydiscard images captured by image sensor 220 that do not include at leasta portion of at least one person in the image and/or do not include atleast a portion of at least one face in the image while retaining imagesthat include at least a portion of at least one person and/or at least aportion of at least one face. For example, facial image module 2001 mayapply image analysis to determine whether a person, a face, and/or aportion thereof are present in the image.

Facial image module 2001 may detect persons and/or faces in receivedimages using any known techniques. For example, facial image module 2001may perform image processing on a currently received image and determinethat the received image corresponds to an image of a person or a facethat has been previously stored in memory. Thus, facial image module2001 may perform pixel-by-pixel matching. In other embodiments, facialimage module 2001 may perform more sophisticated techniques. Forexample, facial image module 2001 may use one or more classifiers (orcascading classifiers) to detect a person or a face included in theimage. Additionally or alternatively, facial image module 2001 may use aneural network to detect a person or a face included in the image.Alternatively, facial image module 2001 may use a neural network toderive a plurality of features (e.g., a feature set) from the image andthen map the derived features (or feature set) onto features associatedwith detection of a person or a face. The mapping may be performed usingsimple correlation or using more sophisticated techniques, such asanother neural network.

In any of the embodiments above, detections may have associatedprobabilities or confidence levels, e.g., output from the neuralnetwork; calculated based on percentage of matching pixels, features,etc.; or the like. Accordingly, in some embodiments, facial image module2001 may finalize a detection of a person or a face only if thedetection has an associated probability above a threshold.Alternatively, facial image module 2001 may apply further detectiontechniques if the detection is indeterminate, e.g., above a firstthreshold but below a second, higher threshold. In such an embodiment,facial image module 2001 may apply a classifier and may, if theclassifier produces a detection with an indeterminate probability, thenapply one or more additional classifiers and/or a neural network toproduce one or more additional detection probabilities. Based on theadditional detection probabilities, e.g., if they exceed the secondthreshold, facial image module 2001 may finalize the detection. Anydetection not finalized may be discarded.

In any of the embodiments above, the image may include more than oneperson and/or more than one face or may include no persons and/or nofaces. Facial image module 2001 may therefore use a bounding boxarchitecture, such as You Only Look Once (YOLO) or Single-Shot Detector(SSD), to identify bounding boxes including a person therein. Forexample, if person identification module 2001 uses a bounding boxarchitecture and receives no bounding boxes classified as persons and/orfaces or receives bounding boxes all having person and/or facialclassifications below a threshold (e.g., less than 50%, less than 40%,less than 25%, less than 10%, less than 5%, or the like), facial imagemodule 2001 may discard the image as including no persons and/or nofaces. Similarly, if facial image module 2001 uses a bounding boxarchitecture and receives a plurality of bounding boxes classified aspersons and/or faces or receives a plurality of bounding boxes havingperson and/or facial classifications above a threshold (e.g., more than50%, more than 40%, more than 25%, more than 10%, more than 5%, or thelike), facial image module 2001 may detect portions of the image definedby the plurality of bounding boxes as including persons and/or faces.

In some embodiments, the images may comprise a first plurality of facialimages of individuals with whom the user interacted on a first calendardate and a second plurality of facial images of individuals with whomthe user interacted on a second calendar date. For example, image sensor220 may time-stamp the generate image such that facial image module 2001receives an associated calendar date with each received image.Alternatively, facial image module 2001 may generate and attach theassociated calendar date to each image upon receipt.

Additionally or alternatively, facial image module 2001 may receive,from an image sensor of the wearable apparatus, at least one imagecaptured in a time window. For example, as explained above, image sensor220 or facial image module 2001 may stamp an image with an associateddate. Accordingly, the time window may comprise one or more days betweenone calendar date and another calendar date, whether countinginclusively of the one calendar date and/or the other calendar date orexclusively of both.

Additionally, image sensor 220 or facial image module 2001 may stamp theimage with an associated time. Accordingly, the time window may comprisea period of time between a start time and an end time. The start timeand end time may comprise times on the same calendar date or maycomprise times on different calendar dates.

In addition to receiving images during a time window, facial imagemodule 2001 may also receive a file of sound captured during the timewindow. The file of sound may also have been captured in a vicinity(e.g., within 5 meters, within 10 meters, within 20 meters, or the like)of image sensor 220 of capturing unit 710. In some embodiments, thesound file may be received from a mobile communications device otherthan the wearable image sensor. For example, a mobile communicationsdevice associated with the same user as the user of the wearableapparatus may record the sound file in a vicinity of image sensor 220 ofcapturing unit 710.

Additionally or alternatively, facial image module 2001 may receivesound data associated with the facial images. For example, the sounddata may be retrieved from a memory where it is stored in associationwith the facial images. Additionally or alternatively, the sound datamay be recorded in a vicinity (e.g., within 5 meters, within 10 meters,within 20 meters, or the like) of image sensor 220 of capturing unit 710and/or in a vicinity (e.g., 5 seconds, 10 seconds, 30 seconds, 1 minute,5 minute, or the like) of the facial images being captured.

In any of the embodiments discussed above, received images or facialimages may vary in size, shape, and content. For example, in someembodiments, an image including a wider scene (e.g., a plurality ofindividuals, or an individual and his or her surrounding environment)may be cropped or otherwise truncated to a face, a portion of a face, aheadshot of a person (e.g., head and shoulders), or a portion of aperson's body including the person's face. In other embodiments,received images may include such wider scenes (e.g., a plurality ofindividuals, or an individual and his or her surrounding environment).In still other embodiments, received images may include a face, aportion of a face, a headshot of a person (e.g., head and shoulders), ora portion of a person's body including the person's face.

Linking attribute module 2002 may be configured to receiving at leastone linking attribute for each of the plurality of facial images. Forexample, linking attribute module 2002 may analyze at least one imagereceived by facial image module 2001 to determine the at least onelinking attribute.

The at least one linking attribute may include an identity of a personappearing in one or more images. In another example, the at least onelinking attribute may include a location. The location may include atleast one of an event, a physical space, and a GPS coordinate. Forexample, the location may include a ‘work retreat’ or ‘concert.’ Inanother example, the geographical location may include a ‘coffee shop’or a ‘neighbor's house.’ In any of the embodiments above, linkingattribute module 2002 may use image analysis, e.g., as described abovewith respect to facial image module 2001, to identify the location.

In another example, the at least one linking attribute may include anoccupation. For example, the location may include ‘contractor,‘teacher,’ or the like. Linking attribute module 2002 may use imageanalysis, e.g., as described above with respect to facial image module2001, to identify the occupation. For example, linking attribute module2002 may identify the occupation by classifying a uniform, a location,one or more tools associated with the occupation, or the like within theimages.

In another example, the at least one linking attribute may include aworkplace. For example, the workplace may include ‘university, ‘123 AStreet,’ or the like. Linking attribute module 2002 may use imageanalysis, e.g., as described above with respect to facial image module2001, to identify the workplace. For example, linking attribute module2002 may identify the workplace by classifying a building, a location,one or more items associated with the workplace, or the like within theimages.

In another example, the at least one linking attribute may include afamily relationship. For example, the family relationship may include‘immediate,’ extended,’ ‘first generation,’ second generation,’‘parents,’ ‘siblings,’ or the like. In yet another example, the at leastone linking attribute may include a social relationship. For example,the social relationship may include ‘close friends,’ ‘friends,’‘acquaintances,’ or the like.

Linking attribute module 2002 may use image analysis, e.g., as describedabove with respect to facial image module 2001, to identify the familyrelationship or social relationship. For example, linking attributemodule 2002 may identify the family relationship or social relationshipby identifying one or more people within the images. The identity of theuser may also be used to determine the family relationship or socialrelationship.

Additionally or alternatively, the at least one linking attribute may bebased on user input. For example, a user may input one or more wordscomprising the at least one linking attribute. In some embodiments,linking attribute module 2002 may present a user with a pick list oflinking attributes.

In embodiments where facial image module 2001 also receives a file ofsound, linking attribute module 2002 may process the sound file toidentify at least one word in the sound file. For example, the at leastone word may be parsed using any audio analysis technique, such aslimited vocabulary recognition, large vocabulary recognition, with orwithout speaker separation, recognition, verification, or the like. Insome embodiments, linking attribute module 2002 may transcribe the soundfile in order to parse the at least one word therefrom.

Additionally or alternatively, linking module 2002 may extract the atleast one word based on a frequency of use associated with each word inthe sound file. For example, linking attribute module 2002 may eliminatestop words or other common words such as articles (‘the,’ ‘a,’ ‘an,’ orthe like), prepositions (‘of,’ ‘from,’ ‘to,’ or the like), or the likefrom the parsed at least one word.

Additionally with or alternatively to receiving at least one linkingattribute, linking attribute 2002 may analyze the plurality of images toidentify contextual information associated with the detected person(e.g., detected by facial image module 2001). For example, thecontextual information may be based, at least in part, on a secondperson appearing in the plurality of images in conjunction with thedetected person. Accordingly, linking attribute module 2002 may useimage analysis, e.g., as described above with respect to facial imagemodule 2001, to identify the second person.

Additionally or alternatively, the contextual information may be based,at least in part, on a uniform the detected person is wearing.Accordingly, linking attribute module 2002 may use image analysis, e.g.,as described above with respect to facial image module 2001, to identifythe uniform.

Additionally or alternatively, the contextual information may be based,at least in part, on a room the detected person appears in. Accordingly,linking attribute module 2002 may use image analysis, e.g., as describedabove with respect to facial image module 2001, to identify the room.

Additionally with or alternatively to receiving at least one linkingattribute, linking attribute 2002 may analyze the plurality of images toidentify a visual context indicator. Accordingly, linking attributemodule 2002 may use image analysis, e.g., as described above withrespect to facial image module 2001, to identify the visual contextindicator.

Association module 2003 may be configured to associate in memory each ofthe plurality of facial images with the at least one linking attributereceived for a corresponding one of the plurality of facial images. Theat least one linking attribute may be determined using image analysisand/or user input, as explained above with respect to linking attributemodule 2002.

In some embodiments, association module 2003 may store a measure ofoccurrences of linking attributes associated with each individual'sfacial image. For example, the measure of occurrences may comprise anumber of times the linking attributes appear in images. The number oftimes may be averaged over a number of images and/or over a period oftime in order to obtain the measure of occurrences. Additionally oralternatively, the measure of occurrences may comprise a measure ofstrength of the linking attributes, e.g., based on sizes within images,prominence within images, or the like.

In embodiments where linking attribute module 2002 processes the soundfile to identify at least one word in the sound file, association module2003 may store in memory an association between the word and the objectimage. Similarly, in embodiments where linking attribute module 2002processes sound data associated with the facial images to extract keywords, association module 2003 may store the key words in memoryassociated with a date of recording.

In embodiments where facial image module 2001 receives a first pluralityof facial images of individuals with whom the user interacted on a firstcalendar date and a second plurality of facial images with whom the userinteracted on a second calendar date, association module 2003 may storean indicator of the first calendar date in association with each of thefirst plurality of individuals and store an indicator of the secondcalendar date in association with each of the second plurality ofindividuals. For example, the indicators may be stored such that theyare retrievable when an individual in the first plurality of individualsor the second plurality of individuals, respectively, are searched for.Additionally or alternatively, the indicators may be indexed such thatthe first plurality of facial images of individuals or the secondplurality of facial images of individuals are retrievable when the firstcalendar date or the second calendar date, respectively, are searchedfor.

In embodiments where the linking attribute module 2002 identifiescontextual information associated with the detected person, associationmodule 2003 may select at least one record associated with the detectedperson and update the at least one record based on the contextualinformation. For example, the identified contextual information may bestored in the record such that it is retrievable when the detectedperson is searched for. Additionally or alternatively, the identifiedcontextual information may be indexed such that the detected person isretrievable when the contextual information is searched for.

In embodiments where the linking attribute module 2002 identifies avisual context indicator, association module 2003 may determinecontextual category based on the detected person and associate thevisual context indicator with the determined contextual category. Forexample, the contextual category may be a work related category, asocial category, or a family category. Additionally or alternatively,association module 2003 may determine a context of a room or a locationbased on one or more persons identified in the room.

Attribute module 2002 may store the visual context indicator with thedetermined contextual category such that the indicator is retrievablewhen the contextual category is searched for. Additionally oralternatively, the visual context indicator may be indexed such that thecontextual category is retrievable when the visual context indicator issearched for.

Display module 2004 may be configured to, for a selected linkingattribute, present on a screen of a display a set of the plurality offacial images that share the selected linking attribute. For example,the set of the plurality of facial images may be arranged in any waydescribed above with respect to visualization module 1704. Accordingly,the set of the plurality of facial images may be displayed as a facecloud

In some embodiments, during presenting, facial images with a highernumber of occurrences of a common linking attribute may be displayedwith a prominence greater than facial images of individuals with a lowernumber of occurrences of a common linking attribute. For example,greater prominence may be reflected in a presented size of facialimages, as depicted above with respect to FIG. 18A. Additionally oralternatively, greater prominence may be reflected in a proximity to afocal location on the display, as depicted above with respect to FIGS.18C-18F.

In embodiments where linking attribute module 2002 presents a user witha pick list of linking attributes, display module 2004 may alter apresentation of facial images on the display in response to a selectionfrom the pick list. For example, display module 2004 may add facialimages to the display and/or remove facial images from the display. Inembodiments where facial images with a higher number of occurrences of acommon linking attribute are displayed with a greater prominence,display module 2004 may modify prominences of the facial images based onthe selection.

In embodiments where linking attribute module 2002 processes a soundfile to identify at least one word in the sound file, display module2004 may receive, at a time remote from the time window, a search inputof the at least one key word. For example, display module 2004 maygenerate a text box to receive the search input. Additionally oralternatively, display module 2004 may generate a list of keywords anddisplay the list of keywords to the user such that the user may inputthe search by selecting one or more keywords from the list.

In such embodiments, display module 2004 may access the memory to locatethe association between the at least one key word and the at least oneobject image and cause a display of the object image on a mobilecommunications device of the user, separate from the wearable imagesensor. Accordingly, display module 2004 may allow for rapid andefficient searches of images obtained from a wearable apparatus. Inaddition, display module 2004 may allow for display of the results on amobile communications device separate from the wearable image sensor,which comprises a non-generic and non-conventional combination ofcomponents for providing search results. The efficient search and uniquecombination of components may improve the user's experience.

In such embodiments, causing the display may include displaying aplurality of facial images each associated with the keyword. Forexample, as explained above, the set of the plurality of facial imagesmay be arranged in any way described above with respect to visualizationmodule 1704. Accordingly, the set of the plurality of facial images maybe displayed as a face cloud.

In embodiments where facial image module 2001 receives a first pluralityof facial images of individuals with whom the user interacted on a firstcalendar date and a second plurality of facial images of individualswith whom the user interacted on a second calendar date, display module2004 may receive a first input request by the user for recall ofinteractions on the first calendar date. For example, display module2004 may display a calendar to the user, and the first input may be aresult of a user's respective touch interactions with screen regionscorresponding to the first calendar date. In response to the firstinput, display module 2004 may display to the user images of at leastsome of the first plurality of individuals.

Additionally or alternatively, display module 2004 may receive a secondinput request by the user for recall of interactions on the secondcalendar date. Similar to the first input, display module 2004 maydisplay a calendar to the user, and the second input may be a result ofa user's respective touch interactions with screen regions correspondingto the second calendar date. In response to the second input, displaymodule 2004 may display to the user images of at least some of thesecond plurality of individuals.

The display of images of at least some of the first plurality ofindividuals and/or images of at least some of the second plurality ofindividuals may be arranged in any way described above with respect tovisualization module 1704. Accordingly, the set of the plurality offacial images may be displayed as a face cloud.

In any of the embodiments above, display module 2004 may additionally oralternatively present to the user the key words associated withinteractions on the first calendar date and the second calendar date.For example, the key words may have been extracted from a sound file (orsound data associated with the facial images) by linking attributemodule 2002. In addition, the key words may have been associated withthe first calendar date and the second calendar date based on a calendardate at which the sound from which the key word was extracted wasrecorded.

Additionally or alternatively, display module 2004 may receive a searchinput by the user, the search input including at least one of the keywords. For example, as explained above, display module 2004 may generatea text box to receive the search input. Additionally or alternatively,display module 2004 may generate a list of keywords and display the listof keywords to the user such that the user may input the search byselecting one or more keywords from the list. In such embodiments, theat least some of the first plurality of individuals and the at leastsome of the second plurality of individuals may be associated with theat least one of the key words.

In any of the embodiments above, display may occur on a mobilecommunications device wirelessly paired to the wearable image sensor.For example, display module 2004 may generate any of the user interfacesdescribed above and transmit the generated interfaces to the mobilecommunications device.

Display module 2004 may also be configured to enable interaction withany of the displays above. For example, display module 2004 may beconfigured to enable a user to select a particular facial image to be afocal point of the face cloud. In such an example, display module 2004may allow a user to click, tap, or otherwise select the particularfacial image. In such embodiments, display module 2004 may receive adesignation of a focal point individual such that the input causes arearrangement of the face cloud while maintaining the focal pointindividual.

FIG. 21A illustrates an example of a database 2100 for indexing facialimages to linking attributes consistent with the present disclosure. Asdepicted in FIG. 21A, a plurality of facial images (e.g., images 2101 a,2101 b, 2101 c, and 2101 d) are stored in the database.

The facial images are indexed to one or more linking attributes for fastretrieval. In the example of FIG. 21A, facial images 2101 a, 2101 b,2101 c, and 2101 d are indexed by first linking attributes 2103 a, 2103b, 2103 c, and 2103 d, respectively, along with second linkingattributes 2107 a, 2107 b, 2107 c, and 2107 d. Although the firstlinking attributes are depicted as locations and the second linkingattributes are depicted as social relationships, the linking attributesmay additionally or alternatively comprise an occupation, a workplace, afamily relationship, or another linking attribute.

As further depicted in the example of FIG. 21A, first linking attributes2103 a, 2103 b, 2103 c, and 2103 d may be indexed to correspondingoccurrences 2105 a, 2105 b, 2105 c, and 2105 d, and second linkingattributes 2107 a, 2107 b, 2107 c, and 2107 d may be indexed tocorresponding occurrences 2109 a, 2109 b, 2109 c, and 2109 d. Forexample, occurrences 2105 a, 2105 b, 2105 c, and 2105 d may comprise anumber of times a person depicted in corresponding facial images 2101 a,2101 b, 2101 c, and 2101 d, respectively, was captured in an image by awearable apparatus while at corresponding locations 2103 a, 2103 b, 2103c, and 2103 d, respectively. The occurrences may comprise a total numberof images in which the person was captured or a total number ofinteractions with a user of the wearable apparatus during which one ormore images were captured. Therefore, in FIG. 21A, the person associatedwith facial image 2101 a was either captured in one image while at thecoffee shop or was captured in images during one interaction with theuser at the coffee shop; the person associated with facial image 2101 bwas either captured in two images or was captured in images during twointeractions with the user at the coffee shop; the person associatedwith facial image 2101 c was either captured in four images or wascaptured in images during four interactions with the user at the gym;and the person associated with facial image 2101 d was either capturedin one image or was captured in images during one interaction with theuser at the park.

In another example, occurrences 2109 a, 2109 b, 2109 c, and 2109 d maycomprise a number of times a person depicted in corresponding facialimages 2101 a, 2101 b, 2101 c, and 2101 d, respectively, was captured inan image by a wearable apparatus during a particular time period. Theoccurrences may comprise a total number of images in which the personwas captured or a total number of interactions with a user of thewearable apparatus during which one or more images were captured.Therefore, in FIG. 21A, the person associated with facial image 2101 awas either captured in two images this week or was captured in imagesduring two interactions with the user this week; the person associatedwith facial image 2101 b was either captured in three images this weekor was captured in images during three interactions with the user thisweek; the person associated with facial image 2101 c was either capturedin five images this week or was captured in images during fiveinteractions with the user this week; and the person associated withfacial image 2101 d was either captured in one image this week or wascaptured in images during one interaction with the user this week.Although one week is used as the time period, other time periods such asone day, two weeks, one month, six months, one year, or the like may beused.

FIG. 21B illustrates an example of a database 2120 for indexing soundfiles to extracted words and object images consistent with the presentdisclosure. As depicted in FIG. 21B, a plurality of sound files (e.g.,files 2121 a, 2121 b, and 2121 c) are stored in the database. The soundfiles may have been captured in a vicinity of a wearable image sensorduring a time window. The sounds files may have been captured by amicrophone of the wearable apparatus or by a mobile communicationsdevice other than the wearable image sensor. Additionally oralternatively, one or more words (e.g., words 2123 a, 2123 b, and 2123c) may be stored in the database. The one or more words may be extractedfrom the plurality of sound files, as explained above with respect tolinking attribute module 2002.

As further depicted in FIG. 21B, the database may store a plurality ofobject images (e.g., images 2125 a, 2125 b, and 2125 c). The objectimages may have been captured by the wearable image sensor during thetime window. Although depicted as images of inanimate objects in FIG.21B, the database may include an object image that includes a facialimage or is a facial image. Words 2123 a, 2123 b, and 2123 c may be usedto index images 2125 a, 2125 b, and 2125 c, respectively, for fastretrieval.

FIG. 21C illustrates an example of a database 2140 for indexing facialimages to calendar dates consistent with the present disclosure. Asdepicted in FIG. 21C, a plurality of calendar dates (e.g., dates 2141 aand 2141 b) are stored in the database and used to index a plurality offacial images (e.g., images 2145 a, 2145 b, 2145 c, 2145 d, and 2145 e),for fast retrieval.

Although not depicted in FIG. 21C, database 2140 may further store keywords from sound data associated with the facial images. The stored keywords may be indexed by the stored dates and/or may be used to index thestored facial images.

It will be appreciated that wherever images or sound files are depictedin FIGS. 21A-21C, they may refer to a pointer to an image or a soundfile or another storage unit, or to metadata describing the image orsound file.

FIG. 22A is a flowchart of a method 2200 for indexing facial images tolinking attributes. Method 2200 may be implemented by a general-purposecomputer or a special-purpose computer built according to embodiments ofthe present disclosure. For example, method 2200 may be executed by atleast one of processors 210, 210 a, 210 b, and 540. In some embodiments,the steps of method 2200 may be performed by wearable apparatus 110. Inother embodiments, the steps of method 2200 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, ear phones,etc.).

At step 2201, the processor may receive facial images of a plurality ofindividuals with whom a user interacted. For example, the processor mayreceive the facial images from an image sensor worn by a user (e.g.,image sensor 220 of capturing unit 710).

At step 2203, the processor may determine or receive at least onelinking attribute for each of the plurality of facial images. Forexample, the processor may determine the at least one linking attributeusing image analysis and/or receive the at least one linking attributeas input from a user, as described above with respect to linkingattribute module 2002 of FIG. 20. The at least one linking attribute mayinclude at least one of a location, an occupation, a workplace, a familyrelationship, and a social relationship.

At step 2205, the processor may associate in memory each of theplurality of facial images with the at least one linking attributereceived for a corresponding one of the plurality of facial images. Forexample, the processor may store the facial images indexed by the atleast one linking attribute, as depicted in the example of FIG. 21A.

In some embodiments, the processor may further store a measure ofoccurrences of linking attributes associated with each individual'sfacial image. For example, the processor may store the facial imagesand/or the at least one linking attribute indexed by the occurrences, asdepicted in the example of FIG. 21A.

At step 2207, the processor may, for a selected linking attribute,present on a screen of a display a set of the plurality of facial imagesthat share the selected linking attribute. For example, the processormay generate a face cloud, as explained above with respect tovisualization module 1704 of FIG. 17. Examples of generated face cloudsare also depicted in FIGS. 18A-18H described above.

In embodiments where occurrences are also stored, during presenting,facial images with a higher number of occurrences of a common linkingattribute may be displayed with a prominence greater than facial imagesof individuals with a lower number of occurrences of a common linkingattribute. For example, greater prominence may be reflected in apresented size of facial images, as depicted above with respect to FIG.18A. Additionally or alternatively, greater prominence may be reflectedin a proximity to a focal location on the display, as depicted abovewith respect to FIGS. 18C-18F.

In some embodiments, method 2200 may include additional steps. Forexample, method 2200 may include presenting a user with a pick list oflinking attributes and altering a presentation of facial images on thedisplay in response to a selection from the pick list, as explainedabove with respect to display module 2004 of FIG. 20.

Moreover, any of the interactions described above with respect tovisualization module 1704 of FIG. 17 may be implemented in method 2200.For example, the set of the plurality of facial images may be updatedand/or animated in response to input from the user.

FIG. 22B is a flowchart of a method 2210 for indexing words in soundfiles to object images. Method 2210 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 2210 maybe executed by at least one of processors 210, 210 a, 210 b, and 540. Insome embodiments, the steps of method 2210 may be performed by wearableapparatus 110. In other embodiments, the steps of method 2210 may beperformed by wearable apparatus 110 and one or more external devices(e.g., a processor included in an external server that receives datafrom wearable apparatus 110 over a network and/or a processor includedin an external device such as a laptop, smartwatch, smartphone, tablet,or ear phones, etc.).

At step 2211, the processor may receive from a wearable image sensorworn by a user at least one image captured in a time window. Forexample, the processor may receive images from image sensor 220 ofcapturing unit 710. The image may comprise an object image. In someembodiments, the object image may include a facial image or be a facialimage.

At step 2213, the processor may receive a file of sound captured in avicinity of the image sensor during the time window. For example, amicrophone of a wearable apparatus including the wearable image sensormay capture the file of sound. Additionally or alternatively, a mobilecommunications device other than the wearable image sensor may capturethe file of sound and send the file to the processor.

At step 2215, the processor may process the sound file to identify atleast one word in the sound file. For example, the processor may processthe file as described above with respect to linking attribute module2002.

At step 2217, the processor may store in memory an association betweenthe word and the object image. For example, the processor may store theobject image indexed by the at least one word, as depicted in theexample of FIG. 21B.

At step 2219, the processor may receive at a time remote from the timewindow a search input of the at least one key word. As used herein,“remote” may refer to any time at least a threshold amount of time laterthan the time window, such as at least 1 minute, 5 minutes, 15 minutes,1 hour, 1 day, 1 week, or the like. As explained above with respect todisplay module 2004, the processor may generate a text box (e.g., fordisplay on the wearable apparatus or on the mobile communicationsdevice) to receive the search input and/or may generate a list ofkeywords and display the list of keywords to the user (e.g., via thewearable apparatus or on the mobile communications device) such that theuser may input the search by selecting one or more keywords from thelist.

At step 2221, the processor may access the memory to locate theassociation between the at least one key word and the at least oneobject image. For example, the processor may search indexed keywords toobtain at least one object image stored in a database and indexed to theat least one key word.

At step 2223, the processor may cause a display of the object image on amobile communications device of the user, separate from the wearableimage sensor. For example, the processor may transmit the object imageto the mobile communications device for display.

In some embodiments, the display may occur on a mobile communicationsdevice wirelessly paired to the wearable image sensor. For example, themobile communications device may be paired to the wearable sensor via aradio frequency, such as Bluetooth®, or via a computer network, such asthe Internet.

In some embodiments, causing the display may include displaying aplurality of facial images each associated with the keyword. Forexample, the processor may generate a face cloud, as explained abovewith respect to visualization module 1704 of FIG. 17. Examples ofgenerated face clouds are also depicted in FIGS. 18A-18H, describedabove.

Moreover, any of the interactions described above with respect tovisualization module 1704 of FIG. 17 may be implemented in method 2220.For example, the object image (or the plurality of facial images) may beupdated and/or animated in response to input from the user.

FIG. 22C is a flowchart of a method 2230 for indexing facial images tocalendar dates. Method 2230 may be implemented by a general-purposecomputer or a special-purpose computer built according to embodiments ofthe present disclosure. For example, method 2230 may be executed by atleast one of processors 210, 210 a, 210 b, and 540. In some embodiments,the steps of method 2230 may be performed by wearable apparatus 110. Inother embodiments, the steps of method 2230 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, ear phones,etc.).

At step 2231, the processor may receive from a wearable image sensorworn by a user a first plurality of facial images of individuals withwhom the user interacted on a first calendar date. For example, theprocessor may receive the first plurality of facial images from imagesensor 220 of capturing unit 710.

At step 2233, the processor may store an indicator of the first calendardate in association with each of the first plurality of individuals. Forexample, as explained above with respect to FIG. 20, image sensor 220 orfacial image module 2001 may stamp the plurality of images with thefirst calendar date.

At step 2235, the processor may receive from the wearable image sensorworn by the user a second plurality of facial images of individuals withwhom the user interacted on a second calendar date. For example, theprocessor may receive the second plurality of facial images from imagesensor 220 of capturing unit 710.

At step 2237, the processor may store an indicator of the secondcalendar date in association with each of the second plurality ofindividuals. For example, as explained above with respect to FIG. 20,image sensor 220 or facial image module 2001 may stamp the plurality ofimages with the second calendar date. It will be appreciated that a datemay include a day, a month, and a year, or a subset thereof, for examplea month and a year, or just a year, a day of week, or the like.

In some embodiments, the processor may further receive sound dataassociated with the facial images. For example, a microphone of awearable apparatus including the wearable image sensor may capture thesound data. Additionally or alternatively, a mobile communicationsdevice other than the wearable image sensor may capture the sound dataand send the data to the processor.

In embodiments where the processor receives sound data, the processormay further extract key words from the sound data. For example, theprocessor may process the file as described above with respect tolinking attribute module 2002.

In embodiments where the processor extracts key words, the processor mayfurther store the key words in memory associated with a date ofrecording. For example, as explained above with respect to FIG. 20,facial image module 2001 may stamp the sound data with the date ofrecording, which may be the first calendar date or the second calendardate.

At step 2239, the processor may receive a first input request by theuser for recall of interactions on the first calendar date. For example,the processor may generate a text box (e.g., for display on the wearableapparatus or on a mobile communications device other than the wearableimage sensor) to receive the input and/or may display a calendar to theuser (e.g., via the wearable apparatus or on the mobile communicationsdevice) such that the first input is a result of a user's respectivetouch interactions with screen regions corresponding to the firstcalendar date.

At step 2241, the processor may, in response to the first input, displayto the user images of at least some of the second plurality ofindividuals. For example, the processor may generate a face cloud, asexplained above with respect to visualization module 1704 of FIG. 17.Examples of generated face clouds are also depicted in FIGS. 18A-18H,described above.

In embodiments where the processor stores key words, the processor mayfurther present to the user the key words associated with interactionson the first calendar date. For example, the processor may present thekey words in a list or in a graphic, such as a word cloud.

At step 2243, the processor may receive a second input request by theuser for recall of interactions on the second calendar date. Forexample, the processor may generate a text box (e.g., for display on thewearable apparatus or on a mobile communications device other than thewearable image sensor) to receive the input and/or may display acalendar to the user (e.g., via the wearable apparatus or on the mobilecommunications device) such that the second input is a result of auser's respective touch interactions with screen regions correspondingto the second calendar date.

At step 2245, the processor may, in response to the second input,display to the user images of at least some of the first plurality ofindividuals. For example, the processor may generate a face cloud, asexplained above with respect to visualization module 1704 of FIG. 17.Examples of generated face clouds are also depicted in FIGS. 18A-18H,described above.

In embodiments where the processor stores key words, the processor mayfurther present to the user the key words associated with interactionson the second calendar date. For example, the processor may present thekey words in a list or in a graphic, such as a word cloud.

Moreover, any of the interactions described above with respect tovisualization module 1704 of FIG. 17 may be implemented in method 2230.For example, the object image (or the plurality of facial images) may beupdated and/or animated in response to input from the user.

In some embodiments, method 2230 may include additional steps. Forexample, in embodiments where the processor stores key words, method2230 may include receiving a search input by the user, the search inputincluding at least one of the key words. As explained above, theprocessor may generate a text box (e.g., for display on the wearableapparatus or on the mobile communications device) to receive the searchinput and/or may generate a list of keywords and display the list ofkeywords to the user (e.g., via the wearable apparatus or on the mobilecommunications device) such that the user may input the search byselecting one or more keywords from the list. In such embodiments, theat least some of the first plurality of individuals displayed in step2241 and/or the at least some of the second plurality of individualsdisplayed in step 2245 may be associated with the at least one of thekey words.

FIG. 22D is a flowchart of a method 2250 for indexing facial images tocontextual information extracted from images. Method 2250 may beimplemented by a general-purpose computer or a special-purpose computerbuilt according to embodiments of the present disclosure. For example,method 2250 may be executed by at least one of processors 210, 210 a,210 b, and 540. In some embodiments, the steps of method 2250 may beperformed by wearable apparatus 110. In other embodiments, the steps ofmethod 2250 may be performed by wearable apparatus 110 and one or moreexternal devices (e.g., a processor included in an external server thatreceives data from wearable apparatus 110 over a network and/or aprocessor included in an external device such as a laptop, smartwatch,smartphone, tablet, ear phones, etc.).

At step 2251, the processor may analyze a plurality of images to detectat least one person. For example, the processor may receive theplurality of images from a wearable image sensor configured to capture aplurality of images from an environment of a user of the wearableapparatus (e.g., image sensor 220 of capturing unit 710). The processormay detect the at least one person using image analysis, e.g., asdescribed above with respect to facial image module 2001.

At step 2253, the processor may select at least one record associatedwith the detected person. For example, the processor may identify thedetected at least one person using image analysis, e.g., as describedabove with respect to person identification module 1701. The at leastone record may be selected based on the identification.

At step 2255, the processor may analyze the plurality of images toidentify contextual information associated with the detected person. Theprocessor may identify the contextual information using image analysis,e.g., as described above with respect to linking attribute module 2002.

Contextual information may be based, at least in part, on a secondperson appearing in the plurality of images in conjunction with thedetected person. For example, the processor may identify the secondperson using image analysis, e.g., as described above with respect toperson identification module 1701. The contextual information maycomprise a relationship determined between the detected person and thesecond person, such as ‘family,’ friend,’ ‘father-son,’ or the like.Additionally or alternatively, the contextual information may comprise adetermined mood of the detected person toward the second person and/or adetermined mood of the second person toward the detected person. As anexample, the contextual information may comprise ‘exasperated’ if thedetected person rolls her eyes at the second person, may comprise‘angry’ if the second person has furrowed brows and/or is yelling at thedetected person, or the like.

Additionally or alternatively, contextual information may be based, atleast in part, on a uniform the detected person is wearing. For example,if the detected person is wearing a police uniform, the contextualinformation may comprise ‘police,’ ‘police officer,23 or the like. Inanother example, if the detected person is wearing a pilot uniform, thecontextual information may comprise ‘pilot’ or the like.

Additionally or alternatively, contextual information may be based, atleast in part, on a room the detected person appears in. For example, ifthe detected person is in a gym, the contextual information may comprise‘gym,’ workout,' or the like. In another example, if the detected personis in an office, the contextual information may comprise ‘office,’’coworker,’ or the like.

Any of the examples above may be combined. For example, if the room thedetected person appears in is a conference room and the second person isidentified as a boss, the contextual information may comprise‘conference meeting,’ ‘important meeting,’ or the like. In anotherexample, if the room the detected person appears in is a police stationand the uniform is a police uniform, the contextual information maycomprise ‘crime report’ or the like. In yet another example, if theuniform is a business suit, and the second person is identified as aCEO, the contextual information may comprise ‘business meeting,’ clientmeeting,’ ‘board meeting,’ ‘or the like.

At step 2257, the processor may update the at least one record based onthe contextual information. For example, the processor may store thecontextual information in association with the at least one recordand/or may index the at least one record by the contextual information.

FIG. 22E is a flowchart of a method 2260 for indexing visual contextindicators extracted from images to contextual categories associatedwith persons. Method 2260 may be implemented by a general-purposecomputer or a special-purpose computer built according to embodiments ofthe present disclosure. For example, method 2260 may be executed by atleast one of processors 210, 210 a, 210 b, and 540. In some embodiments,the steps of method 2260 may be performed by wearable apparatus 110. Inother embodiments, the steps of method 2260 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, ear phones,etc.).

At step 2261, the processor may analyze a plurality of images to detectat least one person. For example, the processor may receive theplurality of images from a wearable image sensor configured to capture aplurality of images from an environment of a user of the wearableapparatus (e.g., image sensor 220 of capturing unit 710). The processormay detect the at least one person using image analysis, e.g., asdescribed above with respect to facial image module 2001.

At step 2263, the processor may determine a contextual category based onthe detected person. For example, the processor may identify thedetected at least one person using image analysis, e.g., as describedabove with respect to person identification module 1701. The contextualcategory may be selected based on the identification.

The contextual category may be work related. For example, the contextualcategory may comprise an event, such as ‘meeting,’ conference; or thelike; a person, such as ‘coworker, ‘boss,’ or the like; a location, suchas ‘office,’ breakroom; or the like; etc.

Additionally or alternatively, the contextual category may be a socialcategory. For example, the contextual category may comprise an event,such as ‘coffee,’ lunch; or the like; a person, such as ‘close friend,‘acquaintance,’ or the like; a location, such as ‘coffee shop,’ subway;or the like;

etc.

Additionally or alternatively, the contextual category may be a familycategory. For example, the contextual category may comprise an event,such as ‘dinner,’ family vacation,' or the like; a person, such as‘parent, ‘sibling,’ or the like; a location, such as ‘home,’ ‘grandma'shouse,’ or the like; etc.

Additionally or alternatively, the contextual category may be based on acontext of a room or a location based on one or more persons identifiedin the room. For example, the room may be identified as the bedroom suchthat a family category is selected, the location may be identified as anoffice building such that a work related category is selected, or thelike.

At step 2265, the processor may analyze the plurality of images toidentify a visual context indicator. The processor may identify thevisual context indicator using image analysis, e.g., as described abovewith respect to facial image module 2001.

The visual context indicator may comprise an object, a person, or agesture associated with the contextual category. For example, a stapler,a boss, or typing may all be associated with a work related contextualcategory. In another example, tickets to a sporting game or event, afriend, or laughing may all be associated with a social category. In yetanother example, a toy, a sibling, or hugging may all be associated witha family category.

At step 2267, the processor may associate the visual context indicatorwith the determined contextual category. For example, the processor maystore the visual context indicator in association with the determinedcontextual category. Additionally or alternatively, the processor mayindex the visual context indicator by the determined contextual categoryand/or may index the determined contextual category by the visualcontext indicator.

Automatic Object Comparison

In some embodiments, a wearable apparatus of the present disclosure mayprovide for automatic object comparison. For example, the apparatus mayautomatically identify two objects, look-up information related to theidentities of the two objects, and display the comparison such that theuser may ascertain differences between the two objects. By performingthe look-up, the wearable apparatus may provide for a faster and moreaccurate comparison as compared with manual, subjective forms ofcomparison.

In addition, rather than using conventional user interfaces, embodimentsof the present disclosure instead use specific triggers to automaticallyperform the comparison. Conventional interfaces are generally text-basedrather than trigger-based and therefore difficult to use for fast andautomatic comparison. Accordingly, embodiments of the present disclosureprovide for improved user experiences with the wearable apparatus byproviding for easier manipulation by the user.

FIG. 23 illustrates an exemplary embodiment of a memory 2300 containingsoftware modules consistent with the present disclosure. Included inmemory 2300 are object image module 2301, look-up module 2302,comparison module 2303, and display module 2304. Modules 2301, 2302,2303, and 2304 may contain software instructions for execution by atleast one processing device, e.g., processor 210, included with awearable apparatus. In some embodiments, memory 2300 may comprise anon-transitory computer readable medium that resides on a mobilecommunications device (e.g., a smart phone) configured to be wirelesslypaired with wearable image sensor 220 of capturing unit 710.

Object image module 2301, look-up module 2302, comparison module 2303,and display module 2304 may cooperate to perform automatic objectcomparison, as well as cooperating to allow particular displays of suchcomparisons. Memory 2300 may be separate from and/or integrated withmemory 550 of FIG. 6, described above. Similarly, orientationidentification module 601, orientation adjustment module 602, andmonitoring module 603 of FIG. 6 may operate in tandem or concurrentlywith object image module 2301, look-up module 2302, comparison module2303, and display module 2304 of FIG. 23.

Object image module 2301 may be configured to determine from at leastone of a plurality of images a presence of two differing objectsindicated by a user. For example, the images may be from a wearableimage sensor configured to capture the images from an environment of theuser (e.g., image sensor 220 of capturing unit 710).

In some embodiments, object image module 2301 may discard imagescaptured by image sensor 220 that do not include at least two differingobjects while retaining images that include at least two differingobjects. For example, object image module 2301 may apply image analysisto determine objects present in the image.

Object image module 2301 may detect objects in received images using anyknown techniques. For example, object image module 2301 may performimage processing on a currently received image and determine that thereceived image corresponds to an image of an object that has beenpreviously stored in memory. Thus, object image module 2301 may performpixel-by-pixel matching. In other embodiments, object image module 2301may perform more sophisticated techniques. For example, object imagemodule 2301 may use one or more classifiers (or cascading classifiers)to detect an object included in the image. Additionally oralternatively, object image module 2301 may use a neural network todetect an object included in the image. Alternatively, object imagemodule 2301 may use a neural network to derive a plurality of features(e.g., a feature set) from the image and then map the derived features(or feature set) onto features associated with detection of an object.The mapping may be performed using simple correlation or using moresophisticated techniques, such as another neural network.

In any of the embodiments, detections may have associated probabilities,e.g., output from the neural network; calculated based on percentage ofmatching pixels, features, etc.; or the like. Accordingly, object imagemodule 2301 may finalize a detection of an object only if the detectionhas an associated probability above a threshold. Alternatively, objectimage module 2301 may apply further detection techniques if thedetection is indeterminate, e.g., above a first threshold but below asecond, higher threshold. In such an embodiment, object image module2301 may apply a classifier and may, if the classifier produces adetection with an indeterminate probability, then apply one or moreadditional classifiers and/or a neural network to produce one or moreadditional detection probabilities. Based on the additional detectionprobabilities, e.g., if they exceed the second threshold, object imagemodule 2301 may finalize the detection. Any detection not finalized maybe discarded.

In any of the embodiments, the image may include at least two differingobjects. Object image module 2301 may therefore use a bounding boxarchitecture, such as You Only Look Once (YOLO) or Single-Shot Detector(SSD), to identify bounding boxes including objects. For example, ifobject image module 2301 uses a bounding box architecture and receivesno bounding boxes classified as objects or receives bounding boxes allhaving object classifications below a threshold (e.g., less than 50%,less than 40%, less than 25%, less than 10%, less than 5%, or the like),object image module 2301 may discard the image as including no objects.Similarly, if object image module 2301 uses a bounding box architectureand receives a plurality of bounding boxes such that only a singleobject is present in the image or that all objects in the image areclassified as the same object, object image module 2301 may discard theimage as failing to include at least two differing objects. On the otherhand, if object image module 2301 uses a bounding box architecture andreceives a plurality of bounding boxes classified as objects but not allthe same object or receives a plurality of bounding boxes having objectclassifications above a threshold (e.g., more than 50%, more than 40%,more than 25%, more than 10%, more than 5%, or the like) but not all thesame object, object image module 2301 may detect portions of the imagedefined by the plurality of bounding boxes as including differingobjects.

In any of the embodiments discussed, received images may vary in size,shape, and content. For example, in some embodiments, an image includinga wider scene (e.g., a plurality of individuals, or an individual andhis or her surrounding environment) may be cropped or otherwisetruncated to a face, a portion of a face, a headshot of a person (e.g.,head and shoulders), or a portion of a person's body including theperson's face. In other embodiments, received images may include suchwider scenes (e.g., a plurality of individuals, or an individual and hisor her surrounding environment). In still other embodiments, receivedimages may include a face, a portion of a face, a headshot of a person(e.g., head and shoulders), or a portion of a person's body includingthe person's face. An image may also be truncated to include a portioncontaining an inanimate object, such as a product.

Look-up module 2302 may be configured to perform a look up of the twodiffering objects to ascertain identities of the two differing objects.For example, look-up module 2302 may perform an image search using theimage including each object or one or more portion of the imageincluding the objects, e.g., portions remaining after cropping the imageto only include a bounding box associated with each object. The croppedportion may include a buffer between the bounding box and the cropping.

Additionally or alternatively, an identification produced byclassifiers, neural networks, or any other technique or a combinationthereof from object image module 2301 may be used to perform thelook-up. For example, look-up module 2302 may use one or more strings oftext identifying the objects to perform the look-up in addition to or inlieu of using the image (or a portion thereof) including the objects.

Look-up module 2302 may perform the look-up separately for each of thetwo differing objects. Alternatively, look-up module 2302 may performthe look-up in parallel for each of the two differing objects. As usedherein, “in parallel” refers to multithreading, parallel computing, orany other known technique for performing tasks in parallel even if theunderlying architecture (such as a conventional processor) stillrequires switching between threads during execution.

In some embodiments, the two differing objects may be held in at leastone hand of the user. Accordingly, the look up of the two differingobjects to ascertain identities of the two differing objects may betriggered based on a presence of the two differing objects in the handsof the user, as depicted in FIG. 24A, described below. Additionally oralternatively, performing the look up of the two differing objects toascertain identities of the two differing objects may be triggered basedon the two differing objects being pointed at by the user, as depictedin FIG. 24B, described below.

In any of the embodiments described above with triggers, look-up module2302 and/or image module 2301 may use image analysis to detect thetrigger. Accordingly, one or more classifiers and/or neural networks maybe used to identify the user's hands (or other body parts) and determinea gesture therefrom. For example, the image analysis may generate atrigger when the user's hands are identified and classified as holdingtwo differing objects, when the user's hands (or fingers) are identifiedand classified as pointing at two differing objects, or the like.

Additionally with or alternatively to a gesture-based trigger,performing the look up of the two differing objects to ascertainidentities of the two differing objects may be triggered based onreceiving an input from the user. For example, the input from the usermay include a gesture, a voice command and/or a depressed button. Theinput may be received from the wearable apparatus and/or from a mobilecommunications device associated with the user. In embodiments whereinput is combined with a gesture-based trigger, the input may functionto confirm that the user desires an automatic object comparison to beperformed. For example, look-up module 2302 and/or image module 2301 maydetect the trigger and then send a command (to the wearable apparatusand/or a mobile communications device associated with the user) toprompt the user for confirmation.

In some embodiments, performing the look up of the two differing objectsto ascertain the identities of the two differing objects may includeundertaking Internet searches on at least one image of each of thediffering objects. For example, as explained above, look-up module 2302may perform an Internet image search and/or an Internet text search.

Look-up module 2302 may be further configured to perform a look up ofdescriptive information about the two differing objects. For example,look-up module 2302 may perform an image search using the imageincluding each object or one or more portion of the image including theobjects, e.g., portions remaining after cropping the image to onlyinclude a bounding box associated with each object. The cropped portionmay include a buffer between the bounding box and the cropping.

Additionally or alternatively, the identities produced from the firstlook-up described above may be used to perform the look-up fordescriptive information. For example, look-up module 2302 may use one ormore strings of text comprising the identities to perform the look-upfor descriptive information in addition to or in lieu of using the image(or a portion thereof) including the objects.

Look-up module 2302 may perform the look-up for identities and thelook-up for descriptive information separately, as described above.Alternatively, look-up module 2302 may perform the look-upssimultaneously. For example, an image search and/or use of classifiers,neural networks, or any combination thereof from object image module2301 may produce both identities of the two differing objects as well asdescriptive information thereabout. Additionally or alternatively, oneor more strings of text identifying the objects may be used to performthe look-ups for identities of the two differing objects as well asdescriptive information thereabout.

In some embodiments, performing the look up of the descriptiveinformation about the two differing objects may include undertaking anInternet search of product reviews. For example, as explained above,look-up module 2302 may perform a text search of one or moreInternet-based sources identified as indexing or otherwise includingproduct reviews using the identities of the differing objects.Internet-based sources identified as indexing or otherwise includingproduct reviews may include online stores (such as Amazon®, Wal-Mart®,or the like), social networks (such as Facebook®, Twitter®, or thelike), review aggregators (such as Consumer Reports™, CNET®, or thelike), or the like.

Additionally or alternatively, performing the look up of the descriptiveinformation about the two differing objects may include undertaking anInternet search in one or more social networks. For example, asexplained above, look-up module 2302 may perform a text search of one ormore social networks using the identities of the differing objects.

Additionally or alternatively, performing the look up of the descriptiveinformation about the two differing objects may include undertaking anInternet search of text identified on at least one of the two differingobjects. For example, look-up module 2302 may perform a text search ofone or more Internet-based sources using text identified (e.g., usingoptical character recognition) on the objects. The text on the objectsmay have been identified during classification by image module 2301and/or during the look-up for identities described above.

Additionally or alternatively, performing the look up of the descriptiveinformation about the two differing objects may include undertaking anInternet search of dietary information identified on at least one of thetwo differing objects. For example, look-up module 2302 may perform atext search of one or more Internet-based sources using dietaryinformation identified (e.g., using optical character recognition) onthe objects and/or dietary information obtained during the look-up foridentities described above.

Additionally or alternatively, performing the look up of the descriptiveinformation about the two differing objects may include undertaking anInternet search to determine a price of at least one of the twodiffering objects. For example, as explained above, look-up module 2302may perform a text search of one or more Internet-based sourcesidentified as stores or otherwise including prices using the identitiesof the differing objects. Internet-based sources identified as storesmay include Amazon®, Wal-Mart®, or the like, and Internet-based sourcesidentified as otherwise including prices may include Google® Shopping,Price.com, or the like.

Comparison attribute 2303 may be configured to compare the descriptiveinformation about the two differing objects. Any of the descriptiveinformation obtained by look-up module 2302, such as product ratingsand/or reviews, social network mentions, search engine results, dietaryinformation, prices, or the like, may be used for the comparison. Insome embodiments, comparing the descriptive information about the twodiffering objects includes isolating common attributes and collectinginformation about at least one common attribute of each of the twodiffering objects.

Display module 2304 may be configured to cause a display of theinformation about the comparison of the two differing objects in amanner permitting the user to ascertain differences between the twodiffering objects. For example, the user interface may includenon-overlapping images of each object on a screen, e.g., on oppositesides of the screen. The user interface may further include thelooked-up identities and/or looked-up descriptive information in spatialproximity to the corresponding object. The looked-up identities and/orlooked-up descriptive information may be depicted visually and/ortextually.

In any of the embodiments above, causing the display of the informationabout the comparison of the two differing objects may includetransmitting the information about the comparison of the two differingobjects to a mobile communications device. For example, the mobilecommunications device may be paired to the wearable sensor via a radiofrequency, such as Bluetooth®, or via a computer network, such as theInternet.

FIG. 24A illustrates an example of a trigger for performing a look up oftwo differing objects consistent with the present disclosure. Asdepicted in FIG. 24A, a user 2400 may hold a first object 2401 in onehand and a second object 2403 in another hand in order to triggerautomatic object comparison. Although depicted as using separate hands,user 2400 may hold the two differing objects (e.g., object 2401 andobject 2403) in the same hand.

FIG. 24B illustrates another example of a trigger for performing a lookup of two differing objects consistent with the present disclosure. Asdepicted in FIG. 24B, a user 2430 may point to a first object 2431 andto a second object 2433 in order to trigger automatic object comparison.Although depicted as pointing with her hands, user 2400 may point withany portion of her hands (such as one or more fingers, including athumb). Moreover, although depicted as pointing with separate hands,user 2400 may point with the same hand, e.g., first pointing at object2431 and then at object 2433 within a threshold period of time (such as1 second, 5 seconds, 10 seconds, or the like).

FIG. 24C illustrates an example of a user interface 2460 for performinga look up of two differing objects consistent with the presentdisclosure. Interface 2460 may be displayed on the wearable apparatusand/or on a mobile communications device associated with the user. Inthe example of FIG. 24C, a user of interface 2460 may drag-and-drop(e.g., with a mouse, a finger, or the like) any of stored images ofobjects, such as images 2465, 2467, 2469, 2471, 2473, and 2475, ontoinput 2461 to identify the dragged image as the first object. Inaddition, a user of interface 2460 may drag-and-drop (e.g., with amouse, a finger, or the like) any of stored images of objects, such asimages 2465, 2467, 2469, 2471, 2473, and 2475, onto input 2463 toidentify the dragged image as the second object. The automatic objectcomparison may then proceed based on the dragged images. Althoughdepicted as a standalone process, an input from a user obtained using auser interface (e.g., interface 2460 depicted in FIG. 24C) may be usedinstead to confirm a user's desire to perform automatic objectcomparison based on a detected trigger (e.g., as depicted in FIGS. 24Aand 24B).

In any of the examples depicted in FIG. 24A-24C, the status of a gestureas a trigger and/or use of input may depend on settings selected by auser. For example, a user may select one or more triggers as triggersfor automatic object comparison such that other triggers do notfunction. In such an example, a user may select an option such thatpointing at objects triggers comparison while holding objects does notor vice versa.

Additionally or alternatively, a user may set an option to requireconfirmation before performing automatic object comparison based on oneor more triggers. For example, a user may require confirmation beforeperforming automatic object comparison based on pointing at objectsand/or based on holding objects. In another example, a user may requireconfirmation before performing automatic object comparison based onpointing at objects but not before performing automatic objectcomparison based on holding objects or vice versa.

FIG. 25 is a flowchart of a method 2500 for automatic object comparison.Method 2500 may be implemented by a general-purpose computer or aspecial-purpose computer built according to embodiments of the presentdisclosure. For example, method 2500 may be executed by at least one ofprocessors 210, 210 a, 210 b, and 540. In some embodiments, the steps ofmethod 2500 may be performed by wearable apparatus 110. In otherembodiments, the steps of method 2500 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, ear phones,etc.).

At step 2501, the processor may determine from at least one of aplurality of images a presence of two differing objects indicated by auser. For example, the images may be received by the processor from awearable image sensor configured to capture the images from anenvironment of the user (e.g., image sensor 220 of capturing unit 710).The processor may determine the presence of the two differing objectsusing image analysis, as described above with respect to object imagemodule 2301 of FIG. 23.

At step 2503, the processor may perform a look up of the two differingobjects to ascertain identities of the two differing objects. Forexample, the processor may perform the look-up in response to one ormore triggers and/or user input, as described above with respect tolook-up module 2302 of FIG. 23.

At step 2505, the processor may perform a look up of descriptiveinformation about the two differing objects. For example, the processormay undertake one or more Internet searches, as described above withrespect to look-up module 2302 of FIG. 23. Accordingly, the descriptiveinformation may comprise product reviews associated with the twodiffering objects, social network activity associated with the twodiffering objects, text identified on at least one of the two differingobjects, dietary information identified on at least one of the twodiffering objects, a price of at least one of the two differing objects,or any combination thereof.

In some embodiments, steps 2503 and 2505 may be performed together asone search for each product. In other embodiments steps 2503 and 2505may be performed concurrently, sequentially or in any other manner.

At step 2507, the processor may compare the descriptive informationabout the two differing objects. For example, the processor may isolatecommon attributes and collect information about at least one commonattribute of each of the two differing objects. In one example, a brandof the two differing objects may be isolated as a common attribute whileprices of the two differing objects may be identified as different. Inanother example, calorie counts or other portions of dietary informationon the two differing may be isolated as a common attribute while otherportions of dietary information on the two differing objects may beidentified as different. In another example, product reviews associatedwith the two differing objects may be isolated as a common attributewhile prices of the two differing objects and/or text identified on atleast one of the two differing objects may be identified as different.In another example, product reviews associated with the two differingobjects may be isolated as a common attribute while prices of and/orsocial network activity associated with the two differing objects may beidentified as different. None of the examples given above is mutuallyexclusive.

At step 2509, the processor may cause a display of the information aboutthe comparing of the two differing objects in a manner permitting theuser to ascertain differences between the two differing objects. Forexample, as explained above with respect to display module 2304 of FIG.23, the user interface may include non-overlapping images of each objecton a screen, e.g., on opposite sides of the screen, and may include thelooked-up identities and/or looked-up descriptive information, depictedvisually and/or textually, in spatial proximity to the correspondingobject.

In one example, a logo of a brand and/or text of a name of the brand ofeach object may be displayed near the corresponding object if differentand displayed near the center if the same. In another example, caloriecounts or other portions of dietary information of each object may bedisplayed near the corresponding object if different and displayed nearthe center if the same. The dietary information may be shown textuallyand/or visually (e.g., as a graph such as a bar graph or a pie chart).In another example, product reviews of each object may be displayed nearthe corresponding object if different and displayed near the center ifthe same. The product reviews may be shown textually (e.g., with ratingsand/or excerpts) and/or visually (e.g., a number of stars, a bar graph,or the like). The product reviews may be collated into an overall ratingand/or review and/or a representative rating and/or review may beselected. In another example, social network activity of each object maybe displayed near the corresponding object if different and displayednear the center if the same. The social network activity may bedisplayed textually (e.g., as excerpts from posts and/or as numbersregarding likes, retweets, etc.) and/or visually (e.g., a bar graph oflikes, retweets, etc.). None of the examples given above is mutuallyexclusive. In the discussion above, the term “the same” may refer toabsolute identity, or to a degree of similarity, for example a differentof up to 5%, 10%, 20% or the like in numeric values.

Retrieving and Displaying Key Words from Prior Conversations

In some embodiments, a wearable apparatus of the present disclosure mayprovide for retrieval and display of key words from prior conversationsby a user of the wearable apparatus. For example, the apparatus mayautomatically extract key words from audio recorded within a time windowand index the key words to facial images captured within the timewindow. Accordingly, by constructing and indexing the database accordingto the particular rules of the present disclosure, key words related tointeractions between users and known individuals may be retrieved withgreater speed and accuracy than in conventional systems and may beretrieved automatically with rules rather than manually with subjectivejudgment.

FIG. 26 illustrates an exemplary embodiment of a memory 2600 containingsoftware modules consistent with the present disclosure. Included inmemory 2600 are image processing module 2601, sound processing module2602, memory access module 2603, and display module 2604. Modules 2601,2602, 2603, and 2604 may contain software instructions for execution byat least one processing device, e.g., processor 210, included with awearable apparatus. In some embodiments, memory 2600 may comprise anon-transitory computer readable medium that resides on a mobilecommunications device (e.g., a smart phone) configured to be wirelesslypaired with wearable image sensor 220 of capturing unit 710.

Image processing module 2601, sound processing module 2602, memoryaccess module 2603, and display module 2604 may cooperate for retrievaland display of key words from prior conversations. Memory 2600 may beseparate from and/or integrated with memory 550 of FIG. 6, describedabove. Similarly, orientation identification module 601, orientationadjustment module 602, and monitoring module 603 of FIG. 6 may operatein tandem or concurrently with image processing module 2601, soundprocessing module 2602, memory access module 2603, and display module2604 of FIG. 26.

Image processing module 2601 may be configured to receive, from thewearable image sensor, a facial image of an individual with whom a userof the wearable apparatus interacted in a first interaction during atime window. For example, image processing module 2601 may receiveimages from wearable apparatus 110 captured by image sensor 220 ofcapturing unit 710. As discussed herein, received images or facialimages may vary in size, shape, and content. For example, in someembodiments, an image including a wider scene (e.g., a plurality ofindividuals, or an individual and his or her surrounding environment)may be cropped or otherwise truncated to a face, a portion of a face, aheadshot of a person (e.g., head and shoulders), or a portion of aperson's body including the person's face. In other embodiments,received images may include such wider scenes (e.g., a plurality ofindividuals, or an individual and his or her surrounding environment).In still other embodiments, received images may include a face, aportion of a face, a headshot of a person (e.g., head and shoulders), ora portion of a person's body including the person's face.

Image processing module 2601 may further receive, from the wearableimage sensor, another facial image of the individual during a secondinteraction at a time other than during the time window. Imageprocessing module 2601 may use image processing to determine that theindividual in the second interaction is the individual in the firstinteraction. For example, person identification module 1701 may performimage processing on the other received facial image and determine thatthe other received facial image corresponds to a face of the individualin the facial image received from the first interaction. Thus, imageprocessing module 2601 may perform pixel-by-pixel matching. In otherembodiments, image processing module 2601 may perform more sophisticatedtechniques. For example, image processing module 2601 may use one ormore classifiers (or cascading classifiers) to derive an identity of aperson included in the received facial images. Additionally oralternatively, image processing module 2601 may use a neural network toidentify one or more likely identifications of a person included in thereceived facial images. Alternatively, image processing module 2601 mayuse a neural network to derive a plurality of features (e.g., a featureset) from the received facial images and then map the derived features(or feature set) onto one or more likely identifications. The mappingmay be performed using simple correlation or using more sophisticatedtechniques, such as another neural network. It will be appreciated thatthe term “identification” does not necessarily relate to a true identityof a person, and may also relate to associating a captured image of aperson with a person previously captured by the apparatus. Furthermore,the identification of a particular individual may include, for example,a formal name, a nickname, etc.

In any of the embodiments above, likely matches may have associatedprobabilities, e.g., output from the neural network; calculated based onpercentage of matching pixels, features, etc.; or the like. Accordingly,image processing module 2601 may select an identification of a personfrom a plurality of likely matches by selecting the highest probabilitymatch.

In any of the embodiments above, one or more of the received facialimages may include more than one person. Image processing module 2601may therefore use a bounding box architecture, such as You Only LookOnce (YOLO) or Single-Shot Detector (SSD), to identify bounding boxesincluding a person (or a portion thereof) therein. For example, if imageprocessing module 2601 uses a bounding box architecture and receives aplurality of bounding boxes classified as persons, or receives aplurality of bounding boxes having person classifications above athreshold (e.g., more than 50%, more than 40%, more than 25%, more than10%, more than 5%, or the like), image processing module 2601 mayperform identification on portions of the image defined by the pluralityof bounding boxes.

Sound processing module 2602 may be configured to receive sound datacaptured in a vicinity of the image sensor during at least a part of thetime window. For example, the sound data may have been captured in avicinity (e.g., within 5 meters, within 10 meters, within 20 meters, orthe like) of image sensor 220 of capturing unit 710 and/or in a vicinity(e.g., 5 seconds, 10 seconds, 30 seconds, 1 minute, 5 minute, or thelike) of when the facial image was captured.

Sound processing module 2602 may be further configured to process thesound data to identify at least one key word. For example, the at leastone key word may be parsed using any audio analysis technique, such aslimited vocabulary recognition, large vocabulary recognition, with orwithout speaker separation, recognition, verification, or the like. Insome embodiments, sound processing module 2602 may transcribe the sounddata in order to parse the at least one key word therefrom.

Additionally or alternatively, sound processing module 2602 may extractthe at least one key word based on a frequency of use associated witheach word in the sound data. For example, sound processing module 2602may eliminate common or stop words such as articles (‘the,’ ‘a,’ ‘an,’or the like), prepositions (‘of,’ ‘from,’ ‘to,’ or the like), or thelike from the parsed at least one key word.

In some embodiments, processing the sound data to identify at least onekey word may include accessing a remote server. For example, soundprocessing module 2602 may access the remote server to obtain a list ofcommon words, uniqueness rankings for words included in the sound data,or any combination thereof to assist with extraction of the at least onekey word. Alternatively, sound processing module 2602 may transmit thesound data to the remote server for extraction of the at least one keyword and may thus receive the at least one key word from the remoteserver.

Memory access module 2603 may be configured to store in memory anassociation between the key word and the facial image. For example,memory access module 2603 may store the key word with the facial image(and/or an identity of an individual included in the facial image) suchthat the key word is retrievable when the facial image (or the identityof the individual) is searched for. In some embodiments, memory accessmodule 2603 may store the association in a database. The database may bea relational database, in which the identities serve as an index.Additionally or alternatively, the facial image (and/or an identity ofan individual included in the facial image) may be indexed such that itis retrievable when the key word is searched for.

In some embodiments, storing in the memory the association between thekey word and the facial image may include storing the association in aremote server. For example, the memory may be included in a remoteserver accessible over one or more networks.

Memory access module 2603 may be further configured to access the memoryto locate the at least one key word from the first interaction. Forexample, memory access module 2602 may search by an identity of anindividual in the first interaction to retrieve the at least one keyword. Accordingly, memory access module 2602 may generate a queryincluding the identity to run against an index of identities to retrievethe corresponding at least one key word.

In some embodiments, memory access module 2603 may further be configuredto search stored keywords according to a relationship between the userand the individual, keyword, location, synonym, or subject. For example,a plurality of key words may be associated with the same identity (orfacial image) in the memory. To select a key word for display, memoryaccess module 2603 may use a relationship between the user and theindividual, a keyword included in the plurality of key words, a locationof the first interaction or the second interaction, a synonym of one ormore key words in the plurality of keywords, or a subject of the firstinteraction or the second interaction.

Display module 2604 may be configured to, during the second interaction,cause a display of at least one key word on a display visible to theuser, to thereby remind the user of subject matter of the firstinteraction. The display may be included on the wearable apparatus.Alternatively, the display may be included in a mobile communicationsdevice paired with the wearable apparatus. For example, the mobilecommunications device may be paired to the wearable sensor via a radiofrequency, such as Bluetooth®, or via a computer network, such as theInternet.

FIG. 27A illustrates an example of a database 2700 for indexing keywords to interaction frequencies consistent with the present disclosure.As depicted in FIG. 27A, a plurality of key words (e.g., words 2701 a,2701 b, and 2701 c) are stored in the database. In some embodiments, thedatabase may pertain to a particular user. In other embodiments, thedatabase may store data for a plurality of users.

Although not depicted in FIG. 27A, the words may be indexed to facialimages and/or identities of individuals extracted from facial images forfast retrieval. As further depicted in FIG. 27A, the words 2701 a, 2701b, and 2701 c are further indexed by interaction frequency levels 2703a, 2703 b, and 2703 c, respectively. The interaction frequency level maycomprise a number of interactions between the user and a correspondingindividual. In some embodiments, the interaction frequency may be pertime unit, or normalized across time, e.g., averaged over a day, a week,a month, or the like.

As depicted in FIG. 27A, the number of key words stored for eachindividual may be proportional (whether linearly, quadratically, or thelike) with an interaction frequency level. For example, more keywords(or even a full conversation) may be stored if the interaction betweenthe user and the individual is identified as frequent. Alternatively,the number of key words may be reduced in accordance with an interactionfrequency level. For example, fewer may be stored for someone with whomthe user meets daily. A reduction in the number of key words may resultin stricter parsing of sound data. For example, sound processing module2602 may require greater uniqueness (e.g., a higher threshold that auniqueness ranking must exceed) of a word before storing it and/orgreater frequency of the word (e.g., a number of times the word appearsin the sound data) before storing it.

FIG. 27B illustrates an example of a database 2750 for indexing keywords to importance levels within the context of the individualsconsistent with the present disclosure. As depicted in FIG. 27B, aplurality of key words (e.g., words 2751 a, 2751 b, and 2751 c) arestored in the database.

Although not depicted in FIG. 27B, the words may be indexed to facialimages and/or identities of individuals extracted from facial images forfast retrieval. As further depicted in FIG. 27A, the words 2751 a, 2751b, and 2751 c are further indexed by importance levels 2753 a, 2753 b,and 2753 c, respectively. The importance level may comprise a numericalrepresentation of an importance within the context of the correspondingindividual based on job title, professional relationship with the user,social relationship with the user, number of mentions and/or followerson one or more social networks, or any combination thereof.

As depicted in FIG. 27B, the number of key words stored for eachindividual may be proportional (whether linearly, quadratically, or thelike) with an importance level. For example, more keywords (or even, asdepicted in FIG. 27B, a full conversation) may be stored if thecorresponding individual is particularly important, e.g., a familymember, a CEO, a celebrity, or the like. An increase in the number ofkey words may result in looser parsing of sound data. For example, soundprocessing module 2602 may require lesser uniqueness (e.g., a lowerthreshold that a uniqueness ranking must exceed) of a word beforestoring it and/or lesser frequency of the word (e.g., a number of timesthe word appears in the sound data) before storing it.

FIG. 28 is a flowchart of a method 2800 for retrieving and displayingkey words from prior conversations. Method 2800 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 2800 maybe executed by at least one of processors 210, 210 a, 210 b, and 540. Insome embodiments, the steps of method 2800 may be performed by wearableapparatus 110. In other embodiments, the steps of method 2800 may beperformed by wearable apparatus 110 and one or more external devices(e.g., a processor included in an external server that receives datafrom wearable apparatus 110 over a network and/or a processor includedin an external device such as a laptop, smartwatch, smartphone, tablet,ear phones, etc.).

At step 2801, the processor may receive, from the wearable image sensor,a facial image of an individual with whom a user of the wearableapparatus interacted in a first interaction during a time window. Forexample, the image may be from image sensor 220 of capturing unit 710.The processor may determine an identity of the individual using imageanalysis, as described above with respect to image processing module2601 of FIG. 26.

At step 2803, the processor may receive sound data captured in avicinity of the image sensor during at least a part of the time window.For example, a microphone of a wearable apparatus including the wearableimage sensor may have captured the sound data. In such an example, thesound data may be obtained from a sound sensor within a housing thatincludes the wearable image sensor. Additionally or alternatively, amobile communications device other than the wearable image sensor mayhave captured the sound data.

At step 2805, the processor may process the sound data to identify atleast one key word.

For example, the processor may use sound processing techniques asdescribed above with respect to sound processing module 2602 of FIG. 26.

At step 2807, the processor may store in memory an association betweenthe key word and the facial image. For example, the processor may indexthe key word by the facial image or an identity of an individualincluded in the facial image, as described above with respect to memoryaccess module 2603 of FIG. 26.

In some embodiments, at least a portion of the sound data may be stored,the portion determined in accordance with an interaction frequencylevel. For example, as explained above with respect to the example ofFIG. 27A, the portion may be stored if the interaction frequency levelis below a particular threshold (e.g., only for interactions that areuncommon for the user). Alternatively, the portion may be stored if theinteraction frequency level is above a particular threshold (e.g., onlyfor frequent interactions for the user).

In some embodiments, at least a portion of the sound data is stored, theportion determined in accordance with an importance level of theindividual. For example, as explained above with respect to the exampleof FIG. 27B, the portion may be stored if the importance level is abovea particular threshold (e.g., only for individuals identified asparticularly important).

In some embodiments, further information may be stored, such asimportance level, frequency, or the like.

At step 2809, the processor may receive, from the wearable image sensor,another facial image of the individual during a second interaction at atime other than during the time window. At step 2811, the processor mayuse image processing to determine that the individual in the firstinteraction is the individual in the second interaction. For example,the image may be from image sensor 220 of capturing unit 710. The imageanalysis used may be as described above with respect to image processingmodule 2601 of FIG. 26.

At step 2813, the processor may access the memory to locate the at leastone key word from the first interaction. For example, the processor mayrun a query including an identity of the individual against an index ofidentities to retrieve the at least one key word, as described abovewith respect to memory access module 2603 of FIG. 26.

At step 2815, the processor may, during the second interaction, cause adisplay of at least one key word on a display visible to the user, tothereby remind the user of subject matter of the first interaction. Asexplained with respect to display module 2604 of FIG. 26, the displaymay be included on the wearable apparatus or in a mobile communicationsdevice paired with the wearable apparatus.

Method 2800 may further include additional steps. For example, method2800 may include searching stored keywords according to a relationshipbetween the user and the individual, keyword, location, synonym, orsubject, as explained above with respect to memory access module 2603.For example, a plurality of key words may be associated with the sameidentity (or facial image) in the memory such that the processor may useat least one of a relationship between the user and the individual,keyword, location, synonym, or subject to select a key word for display.

In one example, the processor may select at least one of the pluralityof key words, in accordance with the frequency, such that the wordshaving the highest frequency are selected. In another example, theprocessor may select at least one of the plurality of key wordsassociated with laughter (such as a key word from a joke) if therelationship between the user and the individual is social but mayselect at least one of the plurality of key words associated with a jobtask (such as a key word from a meeting or from an assignment) if therelationship between the user and the individual is professional. Inanother example, the processor may select at least one of the pluralityof key words associated with a keyword input by the user, spoken by theuser, or spoken by the individual (e.g., during the second interaction).Accordingly, the processor may select which of the plurality of keywords to display based on currently spoken words by the user and/or theindividual. In another example, the processor may select at least one ofthe plurality of key words associated with a location of the firstinteraction and/or the second interaction. Accordingly, the processormay select which of the plurality of key words to display based on alocational context of the current interaction. In another example, theprocessor may select at least one of the plurality of key words byeliminating key words that are determined to be synonyms. Additionallyor alternatively, the processor may select at least one of the pluralityof key words that is a synonym with a keyword input by the user, spokenby the user, or spoken by the individual (e.g., during the secondinteraction). In another example, the processor may select at least oneof the plurality of key words that is associated with a subject of thesecond interaction. The processor may determine the subject based onwords spoken during the second interaction, facial expressions and/orgestures of the user and/or the individual during the secondinteraction, a context of the second interaction (such as a location ofthe second interaction), or the like. None of the examples given aboveis mutually exclusive with any other example.

Wearable Apparatus Controlled by Environmental Condition

In some embodiments, wearable apparatus 110 may collect informationrelated to the environment of the user of the wearable apparatus 110.The user may configure certain characteristics of his environment totrigger the wearable apparatus 110 to communicate with a mobilecommunications device associated with the user to execute an action.Thus, the wearable apparatus 110 may be able to execute an action forthe user, without prompting. For example, if the user enters a meetingroom, the apparatus 110 may silence the user's mobile device based oncollected image and/or audio data indicating that the user is in ameeting room, thereby eliminating the need for the user to manually sethis mobile device to silent. In another embodiment, apparatus 110 mayinclude a learning mechanism such that, if the user typically silences amobile communications device upon entering a meeting room, the apparatus110 may initiate the action of silencing the mobile communicationsdevice without the user programming entering a meeting room as a triggerfor silencing the mobile communications device.

Currently, wearable devices or apparatuses may allow a user to controlone or more mobile devices. However, current technology does not executeone or more actions without a user prompt. Specifically, currenttechnology is not capable of analyzing image data for direct andindirect cues indicating one or more environmental conditions of a userand based on those conditions, initiating an action.

As described above, in some embodiments, any one of apparatus 110 orcomputing device 120, via processor 210 or 540, may further process atleast the captured image data to provide additional functionality torecognize environmental conditions and/or other information in thecaptured image data. In some embodiments, apparatus 110 may processaudio data received via a microphone to recognize environmentalconditions. In some embodiments, actions may be taken based on theenvironmental condition, such as controlling a device. For example, ifthe apparatus recognizes that the wearer is in a meeting, then theapparatus might permit the phone paired with the camera to switch tovibrate mode, so that phone calls during the meeting do not cause aninterruption. Other triggers and actions based on environmentalconditions are possible.

An environmental condition, as used herein, may refer to any physicalcharacteristic of the environment of the user. For example, anenvironmental condition may be any one or more of: an object present inthe environment, a classification of an environment based on the typesof objects present, or an activity that the user is engaged in. Anenvironmental condition may be, for example, that another person in theenvironment is speaking or that the user is viewing a video presentationthat may or may not include audio. In some embodiments, theenvironmental condition includes an ambient condition. An ambientcondition may be, for example, lighting level or temperature. Anexemplary environmental condition may include the user of the wearableapparatus entering a meeting room. In another embodiment, anenvironmental condition may include the user of the wearable apparatusexiting a meeting room.

In some embodiments, wearable apparatus 110 may collect informationabout one or more environmental conditions by processing images capturedby image sensor 220. Image sensor 220 may collect direct information,for example, that the user is in a room containing a desk and a chair,and may collect indirect information. Indirect information may be, forexample, that the user is in a cold room, which is inferred by analyzingimages containing one or more people, to determine whether the one ormore people are giving visual clues about the temperature in the room,e.g., shivering.

In some embodiments, a user of the wearable apparatus 110 may configurean environmental condition as a trigger associated with an action. Thus,when apparatus 110 detects one or more environmental conditions that aretriggers, apparatus 110 may communicate with a mobile communicationsdevice associated with the user to execute the action associated withthe trigger. In some embodiments, the user may receive a text or anothernotification on his mobile communications device indicating that theapparatus 110 executed the action associated with the trigger. In someembodiments, the user may execute an action, not stored in associationwith a trigger, every time a certain environmental condition is detectedby image sensor 220. If the user executes this same action a thresholdnumber of times, apparatus 110 may automatically execute the action upondetection of the environmental condition and execute instructionscausing the mobile communications device to prompt the user to store theenvironmental condition and action such that the apparatus 110 executesthe action each time the environmental condition is detected.

FIG. 29 illustrates an exemplary embodiment of a memory 2900 containingsoftware modules consistent with the present disclosure. Included inmemory 2900 are an image processing module 2901, an audio processingmodule 2905, a trigger module 2902, an action module 2903, and an actiondatabase 2904. Modules 2901, 2902, 2903, 2905 may contain softwareinstructions for execution by at least one processing device, e.g.,processor 210, included with a wearable apparatus. Image processingmodule 2901, audio processing module 2905, trigger module 2902, andaction module 2903 may cooperate to process an image from an imagesensor, e.g., image sensor 220, identify one or more triggers present inthe environment of the user, and execute an action based on the one ormore triggers. Memory 2900 may be separate from and/or integrated withmemory 550 of FIG. 6, described above. Similarly, orientationidentification module 601, orientation adjustment module 602, and motiontracking module 603 of FIG. 6 may operate in tandem or concurrently withimage analysis module 2901, audio processing module 2905, trigger module2902, and action module 2903 of FIG. 29.

Image processing module 2901 may be configured to identify at least oneenvironmental condition from one or more images. In some embodiments,the at least one environmental condition may be identified through theapplication of one or more image classification techniques. For example,at least one image classification technique may be used to classify atleast one feature of an image. In some embodiments, an imageclassification technique may include one or more of image enhancement,edge detection, image analysis, and data extraction. Specific examplesof the methods for identifying at least one environmental condition areexemplary only, and a person of ordinary skill in the art will recognizeother methods for identifying the at least one environmental conditionthat remains consistent with the present disclosure. In some examples,at least one environmental condition may be detected using an objectdetection algorithm, using a neural network trained to detect objectsand/or objects in images, using a machine learning system, using apattern recognition system, and so forth.

In some examples, at least one person may be identified in theenvironment of the user using a facial recognition algorithm, using aneural network trained to detect faces and/or persons in images, and soforth. In other embodiments, the wearable apparatus 110 may capturethermal images, either alone or in combination with visual images, forprocessing by the thermal signature algorithm. Thermal recognition of atleast one person or object may be desirable in implementations in whichthe wearable apparatus 110 is operating in reduced lighting situations.

An environmental condition may be identified, for example, by analyzingone or more images. An environmental condition may be any physicalcharacteristic of the environment of the user. Environmental conditionsmay include, for example, furniture such as a conference tableindicating a conference room, a clock displaying a time, one or morepeople, etc. In some embodiments, an environmental condition may beidentified by identifying text in an image and analyzing the text. Forexample, identifying the text of a sign hanged outside a room andreading “Meeting Room,” and comparing it to predefined texts may providefor identifying an environmental condition being a meeting. In someembodiments, environmental conditions may include ambient conditions,such as light level and temperature. For example, the image processormay identify a person appearing to be cold, thus indicating a lowtemperature as an ambient condition. Image processing module 2901 mayadditionally identify other objects in the environment of the user, forexample, a book, a newspaper, a computer screen, etc. and associate theobject with a certain task such as reading or viewing a presentation.Image processing module 2901 may also have a machine analysis algorithmincorporated such that a library of known environmental conditions maybe updated each time image processing module 2901 may be used.

In some embodiments, audio processing module 2905 may be configured toidentify at least one environmental condition from audio input receivedvia a microphone of the apparatus 110. In some embodiments, the at leastone environmental condition may be identified through the application ofone or more audio classification techniques. For example, at least oneaudio classification technique may be used to classify speech by thewearer of apparatus 110 or by a person other than the wearer ofapparatus 110. In some embodiments, an audio classification techniquemay include one or more of audio enhancement, natural languageprocessing, speech recognition, and data extraction. For example,recognizing specific words related to the situation, e.g., a meeting,may be useful in identifying an environmental condition and takingassociated actions. Specific examples of the methods for identifying atleast one environmental condition are exemplary only, and a person ofordinary skill in the art will recognize other methods for identifyingthe at least one environmental condition that remains consistent withthe present disclosure. Audio processing module 2905 may also have amachine analysis algorithm incorporated such that a library of knownenvironmental conditions may be updated each time audio processingmodule 2905 may be used.

Further, in some embodiments, apparatus 110 may identify at least oneenvironment condition based on analysis of one or more images incombination with audio data. For example, in such embodiments, imageprocessing module 2901 and/or audio processing module 2905 may work inconjunction to identify the at least one environmental condition. As anexample, image processing module 2901 may identify a conference tableand one or more people in the environment of the user. Additionally,audio processing module 2905, may detect audio data indicative of aperson speaking in the vicinity of the user. In this example, anenvironmental condition may be that the user is in a meeting, based onanalysis of audio and image data.

Trigger module 2902 may be configured to identify one or moreenvironmental conditions in an image, as identified by image processingmodule 2901, as being associated with a particular action to be executedby the processor 210. In some embodiments, in addition to image data,trigger module 2902 may be configured to identify one or moreenvironmental conditions in audio data, as identified by audioprocessing module 2905, as being associated with a particular action tobe executed by processor 210. Triggers may be preconfigured and/ordefined by the user and stored in a trigger database 2904. For example,a user may configure triggers via a mobile application on a mobilecommunications device. Trigger module 2902 may also comprise learningcapabilities, such that learned behaviors are adopted as triggers. Forexample, if a user is always silencing his phone at a particularlocation, the location will be adopted as a trigger for silencing thephone for the user. In some embodiments, if, for example, when imageprocessing module 2901 identifies one or more objects in the field ofview, and each time that set of one or more objects is detected, theuser executes an action, processor 210 may store the one or more objectsand action as a trigger-action in trigger database 2904. In someembodiments, the number of times a trigger-action is executed before itis stored in trigger database 2904 is a preconfigured numerical value.In other embodiments, the user may modify the value.

Trigger database 2904 may store associations between triggers andactions. In some embodiments, a trigger may be a single environmentalcondition. For example, image processing module 2901 may identify, usingfacial recognition or an image processing technique described above, aperson who is sweating in the environment of the user. The presence of aperson sweating may be a trigger associated with the action of loweringthe temperature, via a thermostat, in the room occupied by the personand the user. In another example, image processing module 2901 mayidentify an illuminated screen with moving pictures, i.e., a television,in the environment of the user. A broadcasting television may be atrigger for the processor 210 to execute a command to dim or turn offone or more ambient light sources.

In some embodiments, a trigger may be a combination of environmentalconditions. For example, image processing module may identify severalobjects in a room such as chairs, a conference table, and one or morepeople. The existence of these objects in the environment, or forexample a sign in the entrance to a space containing text indicating ameeting room, may indicate that the user is in a meeting room, whichtriggers the processor 210 to silence the user's mobile communicationdevice. In another example, the combination of image data indicatingthat the user is in a meeting room and audio data indicating thatanother person is speaking may trigger the processor 210 to silence theuser's mobile communication device. In another example, a trigger may bethat the ambient lighting in the environment of the user is below apredefined threshold and that a book is present in front of the user,indicating that the user is engaged in reading a book in low light. Thecombination of the low ambient light and the activity of reading textmay be a trigger for the processor 210 to increase the ambient lightinglevel.

Action module 2903 may receive the action associated with the identifiedtrigger and communicate with one or more devices and/or components toexecute the action. For example, processor 210 may determine that anenvironmental condition is predetermined as a trigger for an actionassociated with a mobile communications device wirelessly paired withthe wearable apparatus and cause the mobile communications device totrigger the action in accordance with the environment condition and tothereby cause the action to occur via the mobile communications device.In other embodiments, the action may be associated with one or moredevices, such as a smart home system and/or components thereof, asecurity system, and one or more personal computing devices of the user(e.g., laptop computer, desktop computer, tablet, additional mobilecommunications device, other wearable device, etc.). For example, actionmodule 2903 may communicate directly with a smart home system or smartsecurity system, or may communicate with a smart device associated withthe user through which the system may alter one or more environmentalconditions.

As described above, a trigger may be associated with one or moreactions, such that when trigger module 2902 identifies one or moretriggers in an environment, apparatus 110 executes one or morepredefined actions. The trigger-action associations may be preconfiguredon the apparatus 110 or may be customizable by the user, for example viaa mobile application. The apparatus 110 may execute the one or moreactions by communicating with a device, via any known wireless standard(e.g., wifi, Bluetooth®, etc.), as well as near-field capacitivecoupling, and other short range wireless techniques, or via a wiredconnection.

In one example, a trigger may be associated with the action of silencingthe user's mobile communications device. In this case, action module2903 may send instructions, wirelessly, to the mobile communicationsdevice to change the sound settings to silent. In another example, atrigger may be associated with dimming or strengthening a light near theuser of the apparatus 110. The processor 210 may send instructions to amobile device to communicate with the light and/or lighting system tochanging the light level. In another embodiment, the processor 210 maysend instructions, wirelessly, directly to the light and/or lightingsystem to change the light level.

In some embodiments, the user may configure the parameters of an actionvia a mobile application on a mobile communications device. In anotherembodiment, the action parameters may be preconfigured on the apparatus110. For example, parameters may include a level of brightness ofambient light, a specific volume (e.g., decibel level) of the mobiledevice, a brightness of a screen, lighting level, or a temperature. Inone example, a user may configure a trigger such that if the imageprocessing module 2901 detects a screen in front of the user at a highlevel of brightness, the processor 210 communicates instructions to themobile communications device of the user to dim the screen to the user'spreferred level of brightness. In another embodiment, the same triggerand action may be preconfigured such that the screen brightness isadjusted to a level that decreases eye strain.

Additionally, in some embodiments, the processor may be programmed tooverride the action associated with the mobile communications device. Insome embodiments, the action associated with the mobile communicationsdevice may be overridden according to a context. The context level mayinclude an importance level of an individual. Other exemplary contextsmay be whether an email or other communications indicates an “Urgent”status, local emergency alerts pushed to the mobile communicationsdevice, and/or calendar events marked as important. In one example, auser may set up an emergency/important contact list on his mobilecommunications device such that when an incoming communication isreceived from one of those contacts, the communications device ringseven though the environmental condition of the user's presence in ameeting room triggered the processor 210 to silence the user's mobilecommunications device. In another example, a user may configure anoverriding context to be the type of a received communication, e.g., atelephone call, email, SMS message, video call, notification, etc. Theuser may configure telephonic calls to override a certaintrigger/action, but other communications, such as text message and emailto abide by the configured action.

In some embodiments, the processor may be programmed to restore themobile communications device settings when the user leaves anenvironment associated with a trigger. For example, a user having amobile communications device set to ring at a certain volume may enter ameeting room, which may trigger the processor to silence the user'smobile communications device. When the user leaves the meeting room, theprocessor may communicate with the user's mobile communications deviceto set the device to ring at the certain volume, consistent with thedevice's settings before entering the meeting room.

Modules 2901-2903 and 2905 may be implemented in software, hardware,firmware, a mix of any of those, or the like. For example, if themodules are implemented in software, they may be stored, for example, inmemory 550. However, in some embodiments, any one or more of modules2901-2903 and 2905 may, for example, be stored in processor 540 and/orlocated on server 250, which may include one or more processing devices.Processing devices of server 250 may be configured to execute theinstructions of modules 2901-2903. In some embodiments, aspects ofmodules 2901-2903 and 2905 may include software, hardware, or firmwareinstructions (or a combination thereof) executable by one or moreprocessors, alone or in various combinations with each other. Forexample, modules 2901-2903 and 2905 may be configured to interact witheach other and/or other modules of server 250 and/or a wearable camerasystem to perform functions consistent with disclosed embodiments. Insome embodiments, any of the disclosed modules may each includededicated sensors (e.g., IR, image sensors, etc.) and/or dedicatedapplication processing devices to perform the functionality associatedwith each module.

FIG. 30A is an exemplary scenario in which the system may identify anenvironmental condition such as the user 3000 entering a meeting room3001. The user 3000 may enter a meeting room 3001 wearing an apparatus,e.g., wearable apparatus 110. One or more objects in the meeting room3001, such as conference table 3002, screen 3003, clock 3004, and lamp3005 may be identifiable by image processing module 2901. Additionally,image processing module 2901 may identify one or more ambient conditionsin meeting room 3001 such as the lighting level. Each of these objectsand/or ambient conditions may be an environmental condition classifiedas a trigger for an action associated with the mobile communicationsdevice of the user.

For example, an environmental condition may be that the room the userentered is identified as a meeting room because of the presence of aconference table 3002 and chairs 3006. The identification of a meetingroom as an environmental condition may be a trigger for the processor210 to send instructions to the mobile communications device of the userto execute an action associated with the mobile communications device.For example, the action associated with the mobile communications devicemay include enabling a vibrate mode. In another example, the actionassociated with the mobile communications device may include at leastone of deactivating sounds and reducing a sound level of the mobilecommunications device. In an additional example, the action associatedwith the mobile communications device may be to send an automatedresponse by text message or email to contacts from whom calls or emailswere received by the mobile communications device.

In another example, a trigger may be based on both audio and image data.For example, processor 210 may cause the mobile communications device tobe set to a vibrate mode when the user enters a conference room. Ifaudio processing module 2905 additionally receives audio data indicativeof a single person speaking in the conference room, e.g., giving apresentation, processor 210 may further cause the mobile phone to be setto a silent mode.

Similarly, in another example, when the user exits the meeting room3001, the image processing module 2901 may analyze one or more images todetermine that the conference table 3002 and presentation screen 3003are no longer present in the user environment, therefore theenvironmental condition is no longer valid. Thus, the processor 210 maycommunicate instructions to the mobile communications device to executeone or more actions associated with the mobile communications devicethat are triggered when the user exits a meeting room. For example, theaction associated with the mobile communications device may includediscontinuing a vibrate mode or retrieving a mode employed prior to theuser entering the meeting. In another example, the action associatedwith the mobile communications device may include at least one ofactivating sounds and increasing a sound level of the mobilecommunications device. In another example, if the user's device was setto silent before entering a meeting room and the trigger of entering ameeting room initiated the device to set the volume to silent, uponexiting the meeting room the device may remain silent, consistent withthe settings of the device before entering the meeting room.

In another embodiment, the apparatus 110 may detect a presentation beingshared on screen 3003 and communicate with a device or smart lightingsystem to dim the lamp 3005 in the conference room or to dim the lightsnearest the screen 3003. Additionally, when the presentation is over andthe screen 3003 dims, the apparatus 110 may communicate with a deviceand/or smart lighting system to increase the lighting level in themeeting room 3001 to a default ambient level. In another example, theaction of dimming the lights and/or silencing the user's device may betriggered upon identification, by audio processing module 2905, of soundcoming from one or more speakers alone or in combination with thedetected light emitted from screen 3003. In another example, acombination of low ambient lighting detected by image sensor 220 andaudio data indicating a person speaking may trigger the processor 210 toreduce the volume level of the mobile communications device or silencethe mobile communications device.

FIG. 30B is another exemplary scenario in which the image processingmodule 2901 may identify one or more environmental conditions such as acomputer 3007, a thermostat 3008, and another person 3009 in at leastone image received by image sensor 220. The processor 210 may beprogrammed to identify in the at least one image that a human depictedin the image is overheated, and the trigger causes the mobilecommunications device to send a signal to a thermostat to decreasetemperature. For example, image processing module 2901 may detect, viafacial recognition software or other methods previously described, thatperson 3009 is sweating and therefore is too hot. An environmentalcondition of a person sweating may be a trigger for the processor 210 tosend instructions to a mobile communications device to communicate withthe thermostat 3008 to decrease the temperature in the environment ofthe apparatus wearer 3010 and person 3009. Other image features mayindicate the person 3009 is too hot, for example, if person 3009 isfanning himself or if he removes his blazer. The temperature to whichthermostat 3008 is set may be preconfigured or may be manually set up bythe user via a mobile application.

Similarly, the system may also detect that person 3009 is too cold in atleast one image. The processor 210 may be programmed to identify in theat least one image that a human depicted in the image is cold, and thetrigger causes the mobile communications device to send a signal to athermostat to increase temperature. For example, person 3009 may beshivering or may be wearing a scarf, jacket, and/or gloves, promptingthe processor 210 to communicate with the mobile communications deviceof the user to raise the temperature of the environment via thermostat3008. Additionally or alternatively, audio analysis may indicate thatperson 3009 sneezes a number of times exceeding a threshold, for exampletwo or more times, thus indicating that person 3009 is cold. Thetemperature to which the apparatus adjusts the thermostat 3008 may bepreconfigured or may be pre-set by the wearer 3010 via a mobileapplication. In another embodiment, processor 210 may communicatedirectly with thermostat 3008 via a wireless network.

In another example, audio data from audio processing module 2905 mayindicate that the person 3009 is speaking. The environmental conditionof another person speaking may be a trigger to the apparatus 110 tosilence the mobile communications device. When the person 3009 isfinished speaking, apparatus 110 may restore the mobile communicationsdevice to its previous settings from before the trigger was detected. Inanother example, apparatus 110 may analyze both audio and image data todetermine that, for example, another person is present and that aconversation or meeting is taking place, and therefore apparatus 110 maysilence the mobile communications device. In another example, a sign inor in the vicinity of a room may indicate that it is a meeting room or aconference room, and apparatus 110 may use optical character recognitionto determine the text. The text may be used alone or in connection withother visual and/or audio cues to determine that a meeting is takingplace.

In another example, the processor may be programmed to identify that theuser is engaged in an activity where a higher or lower light level isdesirable, and the trigger causes the mobile communications device tosend a signal to a lighting system to increase or decrease an amount oflight. Image processing module 2901 may detect, as environmentalconditions, the level of ambient lighting and the brightness of thecomputer screen 3007 that the apparatus wearer 3010 is looking at. Thiscombination of environmental conditions may trigger the action ofsending instructions to the mobile communications device to increase ordecrease either one or both of the ambient lighting level, e.g., lamp3011, or the screen to an optimal brightness to reduce eye strain. Otheractivities where a lower light level is desirable may be when the useris watching television or a movie, or certain laboratory activities inwhich a user may handle photosensitive material. In some embodiments,processor 210 may identify the light source nearest to a light emittingobject, such as a television screen, and send instructions to the mobilecommunications device to increase or decrease the light level of onlythat light source. The number of light sources and level of lightadjusted as a result of the presence of a trigger may be configured bythe user or may be preconfigured.

In yet another example, the processor may be programmed to identify thatreading is occurring and ambient light is insufficient, and the triggercauses the mobile communications device to send a signal to a lightingsystem to increase an amount of light. Image processing module 2901 mayidentify an un-illuminated book or text in front of the user as anenvironmental condition. This environmental condition may trigger theprocessor 210 to send instructions to the mobile communications deviceto increase the light level. In some embodiments, the processor 210 maydirectly execute instructions to modify the light level via a smart homeor smart lighting system.

FIG. 31A is a flowchart of a method 3100 for identifying triggers in atleast one image received by image sensor 220 and executing an actionassociated with a mobile communications device of the user according toa disclosed embodiment. Method 3100 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 3100 maybe executed by at least one of processors 210, 210 a, 210 b, and 540. Insome embodiments, the steps of method 3100 may be performed by wearableapparatus 110. In other embodiments, the steps of method 3100 may beperformed by wearable apparatus 110 and one or more external devices(e.g., a processor included in an external server that receives datafrom wearable apparatus 110 over a network and/or a processor includedin an external device such as a laptop, smartwatch, smartphone, tablet,or ear phones, etc.).

At step 3101, the processor may receive at least one image from awearable image sensor, e.g., sensor 220.

At step 3102, the processor may perform image processing of the at leastone image to identify an environmental condition of the user of thewearable apparatus. For example, image processing module 2901 mayreceive image data from image sensor 220 and identify one or moreenvironmental conditions such as objects, people, and/or ambientconditions in the at least one image as described with reference to FIG.29. As previously described, processor 210 may use image classificationtechniques to identify the one or more environmental conditions presentin the at least one image. In some embodiments, an environmentalcondition may be determined based on the difference between one or moresubsequently captured images. For example, one image may capture aconference table and chairs (objects associated with the environmentalcondition of a meeting room) and a subsequent image may not containthose objects, indicating the environmental condition of the user isexiting a meeting room.

At step 3103, the processor may determine if the environmental conditionis a trigger for an action associated with a mobile communicationsdevice wirelessly paired with the wearable apparatus. As previouslydescribed with reference to FIG. 29, trigger database 2904 may store oneor more environmental conditions that are triggers, the associatedactions, and the parameters of the associated actions. Trigger module2902 may access trigger database 2904 to retrieve trigger dataassociated with the environmental conditions identified by imageprocessing module 2901 at step 3102.

At step 3104, the processor may cause the mobile communications deviceto trigger the action to occur via the mobile communications device. Aspreviously described, action module 2903 may cause the processor to sendinstructions, via a wired or wireless connection, to the mobilecommunications device associated with the user. The instructions maycause the mobile communications device to execute an action on itself ormay cause the mobile communications device to further send instructionsto another device and/or system to execute the action. For example, amobile communications device may store one or more applicationsconfigured to control other devices and/or systems. In another example,the processor 210 may be configured to communicate directly with anotherdevice or system without first sending instructions to the mobilecommunications device. For example, the processor 210 may communicatewirelessly with one or more of a smart home system, security system, orpersonal computing device.

FIG. 31B is a flowchart of a method 3110 for identifying triggers basedon audio data and executing an action associated with a mobilecommunications device of the user according to a disclosed embodiment.Method 3110 may be implemented by a general-purpose computer or aspecial-purpose computer built according to embodiments of the presentdisclosure. For example, method 3110 may be executed by at least one ofprocessors 210, 210 a, 210 b, and 540. In some embodiments, the steps ofmethod 3110 may be performed by wearable apparatus 110. In otherembodiments, the steps of method 3110 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, smartwatch, smartphone, tablet, or ear phones,etc.).

At step 3111, the processor may receive audio data from a wearable audioinput device.

At step 3112, the processor may perform audio processing of the audiodata to identify an environmental condition of the user of the wearableapparatus. For example, audio processing module 2905 may receive audiodata from an audio input device, such as a microphone, and identify oneor more environmental conditions such as another person or other peoplespeaking, music, or an audiovisual presentation, as described withreference to FIG. 29. As previously described, processor 210 may useaudio classification techniques to identify the one or moreenvironmental conditions present in the audio data.

At step 3113, the processor may determine if the environmental conditionis a trigger for an action associated with a mobile communicationsdevice wirelessly paired with the wearable apparatus. As previouslydescribed with reference to FIG. 29, trigger database 2904 may store oneor more environmental conditions that are triggers, the associatedactions, and the parameters of the associated actions. Trigger module2902 may access trigger database 2904 to retrieve trigger dataassociated with the environmental conditions identified by audioprocessing module 2905 at step 3112. In some embodiments, a trigger maybe associated with environmental conditions identified by both image andaudio data.

At step 3114, the processor may cause the mobile communications deviceto trigger the action to occur via the mobile communications device. Aspreviously described, action module 2903 may cause the processor to sendinstructions, via a wired or wireless connection, to the mobilecommunications device associated with the user. The instructions maycause the mobile communications device to execute an action on itself ormay cause the mobile communications device to further send instructionsto another device and/or system to execute the action. For example, amobile communications device may store one or more applicationsconfigured to control other devices and/or systems. In another example,the processor 210 may be configured to communicate directly with anotherdevice or system without first sending instructions to the mobilecommunications device. For example, the processor 210 may communicatewirelessly with one or more of a smart home system, security system, orpersonal computing device.

FIG. 31C is a flowchart of a method 3120 for identifying triggers basedon a combination of audio and image data and executing an actionassociated with a mobile communications device of the user according toa disclosed embodiment. Method 3120 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 3120 maybe executed by at least one of processors 210, 210 a, 210 b, and 540. Insome embodiments, the steps of method 3120 may be performed by wearableapparatus 110. In other embodiments, the steps of method 3120 may beperformed by wearable apparatus 110 and one or more external devices(e.g., a processor included in an external server that receives datafrom wearable apparatus 110 over a network and/or a processor includedin an external device such as a laptop, smartwatch, smartphone, tablet,or ear phones, etc.).

At step 3121 a, the processor may receive audio data from a wearableaudio input device. At step 3121 b, the processor may receive image datafrom a wearable image sensor, e.g., image sensor 220. Steps 3121 a and3121 b may occur concurrently or in tandem.

At step 3122, the processor may perform audio processing of the audiodata and image processing of the image data to identify an environmentalcondition of the user of the wearable apparatus. For example, audioprocessing module 2905 may receive audio data from an audio inputdevice, such as a microphone, and identify one or more environmentalconditions such as another person or other people speaking, music, or anaudiovisual presentation, as described with reference to FIG. 29.Concurrently, or in tandem, image processing module 2901 may receiveimage data from image sensor 220, and identify one or more objectsand/or people in the environment of the user. As previously described,processor 210 may use audio and/or image classification techniques toidentify the one or more environmental conditions present in theenvironment based on the audio and image data.

At step 3123, the processor may determine if the environmental conditionis a trigger for an action associated with a mobile communicationsdevice wirelessly paired with the wearable apparatus. As previouslydescribed with reference to FIG. 29, trigger database 2904 may store oneor more environmental conditions that are triggers, the associatedactions, and the parameters of the associated actions. Trigger module2902 may access trigger database 2904 to retrieve trigger dataassociated with the environmental conditions identified from image andaudio data determined by image processing module 2901 and audioprocessing module 2905 at step 3122. In some embodiments, a trigger maybe associated with environmental conditions identified by audio data,image data, or a combination of both audio and image data.

At step 3124, the processor may cause the mobile communications deviceto trigger the action to occur via the mobile communications device. Aspreviously described, action module 2903 may cause the processor to sendinstructions, via a wired or wireless connection, to the mobilecommunications device associated with the user. The instructions maycause the mobile communications device to execute an action on itself ormay cause the mobile communications device to further send instructionsto another device and/or system to execute the action. For example, amobile communications device may store one or more applicationsconfigured to control other devices and/or systems. In another example,the processor 210 may be configured to communicate directly with anotherdevice or system without first sending instructions to the mobilecommunications device. For example, the processor 210 may communicatewirelessly with one or more of a smart home system, security system, orpersonal computing device.

Remotely Identifying a Location of a Wearable Apparatus

In some embodiments, wearable apparatus 110 may be paired with a mobilecommunications device associated with a user. If the user misplaceswearable apparatus 110, the user may initiate an application on a mobilecommunications device to locate apparatus 110. For example, image sensor220 may collect information related to the environment of apparatus 110and may transmit image data and/or location data to the mobilecommunications device paired with the wearable apparatus 110. Wearableapparatus 110 may use image data of its environment to provide audioand/or spoken directions to the user via the mobile communicationsdevice to assist the user in locating apparatus 110. For example, ifapparatus 110 is under a piece of furniture, e.g., a sofa, the apparatus110 may receive image data depicting a low light environment and/or maycapture an image with an obscured portion of a field of view, apparatus110 may process the image data to identify its location and may promptthe user with an audio instruction, such as, “The device is under thesofa,” via the mobile communications device. In some embodiments, if thelocation of apparatus 110 is not able to be determined from image data,i.e., the image sensor 220 is in a dark space and cannot collect imagedata, the apparatus 110 may send GPS coordinates indicating its locationto the mobile communications device.

When a device is misplaced, a user may use a mobile application via asecond device to locate the misplaced device. For example, a user maylocate a misplaced device based on GPS coordinates or proximity to thesecond device. However, current devices are not capable of identifyingtheir exact location with reference to other objects in theirenvironment. Because the misplaced device is not aware of other objectsand/people in its environments, sufficient information is not availableto the device in order to provide precise directions to enable the userto find it.

As described above, in some embodiments, any one of apparatus 110 orcomputing device 120, via processor 210 or 540, may process capturedimage data to provide additional functionality to recognize objects inits environment. Apparatus 110 may be paired with a mobilecommunications device associated with a user via any known wirelessstandard (e.g., WiFi, Bluetooth®, etc.), as well as near-fieldcapacitive coupling, and other short range wireless techniques. Mobilecommunications device may be a smartphone, laptop, tablet, or otherdevice capable of communicating via a wireless network. Apparatus 110may be configured to provide output, for example, viafeedback-outputting unit 230, using one or more of a wirelesstransceiver 530, an audio speaker, a vibrotactile simulator, or adisplay.

In some embodiments, apparatus 110 may receive, from a mobilecommunications device wirelessly paired with the wearable apparatus 110,a command to provide location information for the wearable apparatus.For example, the user of wearable apparatus 110 may execute a command,via a mobile application installed on a mobile communications device,for the apparatus 110 to transmit location information. In response tothe command received wirelessly from the mobile communications device,apparatus 110 may process image data collected by image sensor 220 andmay determine a location of the apparatus 110.

In some embodiments, apparatus 110 may include a motion sensor, e.g., anaccelerometer or gyroscope. The motion sensor may gather informationabout the apparatus' movement. For example, if the apparatus 110 is in apocket of the user, image sensor 220 may not collect any image data.However, the motion sensor may collect data indicative of a human gaitpattern. In this example, mobile communications device may be promptedto alert the user that the apparatus 110 is in a pocket or in a bag thatthe user is carrying.

Apparatus 110 may transmit location information and/or directionsleading from the location of the mobile communications device to thelocation to the apparatus 110. In some embodiments, the mobilecommunications device is configured to provide an audible description ofthe location of the wearable apparatus based on the information aboutthe location of the wearable apparatus. In some embodiments, the audibledescription includes spoken words specifying the location of thewearable apparatus. For example, via a speaker or a visual display, themobile communications device may output to the user instructions ordirections for finding apparatus 110. In some embodiments, the spokenwords include a specification that the wearable apparatus is locatedunder a piece of furniture. For example, mobile communications devicemay output audio, “Device is under some furniture”, or “Device is underthe sofa.” In other embodiments, the spoken words include aspecification that the wearable apparatus is located in a bag or acontainer. In another embodiment, the spoken words may include aspecification that the wearable apparatus is located in a pocket of auser of the mobile communications device. In yet another embodiment, thespoken words may include a specification that the wearable apparatus islocated in a direction relative to the user of the mobile communicationsdevice.

In other embodiments, the mobile communications device is configured toprovide text output of the location of the wearable apparatus based onthe information about the location of the wearable apparatus. In someembodiments, the text may include a specification that the wearableapparatus is located under a piece of furniture. In other embodiments,the text may include a specification that the wearable apparatus islocated in a bag or a container. In another embodiment, the text mayinclude a specification that the wearable apparatus is located in apocket of a user of the mobile communications device. In yet anotherembodiment, the text may specify that the wearable apparatus is locatedin a direction relative to a user of the mobile communications device.

In another embodiment, locating the apparatus 110 may includeestablishing a wireless pairing between a wearable apparatus and amobile communications device. The mobile communications device may beconfigured for receiving, in a situation when the wearable apparatus ismisplaced, a user input for pinging the wearable apparatus, for example,via an application and/or I/O device of the mobile communicationsdevice. In response to the user input, locating the misplaced apparatusmay include wirelessly transmitting from the mobile communicationsdevice a ping to the wearable apparatus. The ping may include aninstruction for the wearable apparatus to capture at least one image ofsurroundings of the wearable apparatus. The apparatus 110 may analyzethe at least one image to determine a location of the wearable apparatusand cause the location information to be transmitted to the mobilecommunications device.

In another embodiment, apparatus 110 may be configured to use audioinformation in addition to image information. For example, apparatus 110may detect, via a microphone, the sound of a train. Apparatus 110 mayuse this audio information instead of, or in addition to, imageinformation to communicate instructions to the mobile communicationsdevice that apparatus 110 may be at a train station. In another example,apparatus 110 may be in a bag, restricting data the image sensor 220 maycapture. In this example, apparatus 110 may use audio data to determine,for example, that the user is in the same room as the apparatus by usingvoice recognition.

FIG. 32 illustrates an exemplary embodiment of a memory 3200 containingsoftware modules consistent with the present disclosure. Included inmemory 3200 are an image processing module 3201, an audio processingmodule 3204, an apparatus location module 3202, and a direction module3203. Modules 3201, 3202, 3203, 3204 may contain software instructionsfor execution by at least one processing device, e.g., processor 210,included with a wearable apparatus. In some embodiments, any one or moreof image processing module 3201, audio processing module 3204, apparatuslocation module 3202, and direction module 3203 may be executed by amobile communications device wirelessly paired with the wearableapparatus. Image processing module 3201, audio processing module 3204,apparatus location module 3202, and direction module 3203 may cooperateto process one or more images from an image sensor, e.g., image sensor220, and location data, e.g., GPS coordinates of apparatus 110, togenerate a set of instructions to assist the user in locating apparatus110. Memory 3200 may be separate from and/or integrated with memory 550of FIG. 6, described above. Similarly, orientation identification module601, orientation adjustment module 602, and motion tracking module 603of FIG. 6 may operate in tandem or concurrently with image analysismodule 3201, audio processing module 3204, device location module 3202,and direction module 3203 of FIG. 32.

Image processing module 3201 may be configured to identify one or moreobjects in one or more images received from image sensor 220. In someembodiments, objects or people may be identified through the applicationof one or more image classification techniques. For example, at leastone image classification technique may be used to classify at least onefeature of an image. In some embodiments, an image classificationtechnique may include one or more of image enhancement, edge detection,image analysis, and data extraction. Specific examples of the methodsfor identifying one or more objects are exemplary only, and a person ofordinary skill in the art will recognize other methods for identifyingobjects remain consistent with the present disclosure. In some examples,one or more objects may be detected using an object detection algorithm,using a neural network trained to detect objects and/or objects inimages, using a machine learning system, using a pattern recognitionsystem, and so forth.

In some examples, at least one person may be identified in theenvironment of the user using a facial recognition algorithm, using aneural network trained to detect faces and/or persons in images, and soforth. In other embodiments, the wearable apparatus 110 may capturethermal images, either alone or in combination with visual images, forprocessing by the thermal signature algorithm. Thermal recognition of atleast one person or object may be desirable in implementations in whichthe wearable apparatus 110 is operating in reduced lighting situations.

In some embodiments, audio processing module 3204 may be configured toidentify location information based on audio input received via amicrophone of the apparatus 110. In some embodiments, the locationinformation may be identified through the application of one or moreaudio classification techniques. For example, at least one audioclassification technique may be used to classify speech by a personother than the wearer, traffic noises, railroad noises, human noises,etc. In some embodiments, an audio classification technique may includeone or more of audio enhancement, natural language processing, speechrecognition, and data extraction. Specific examples of the methods foridentifying at least one environmental condition are exemplary only, anda person of ordinary skill in the art will recognize other methods foridentifying the at least one environmental condition that remainsconsistent with the present disclosure. Audio processing module 3204 mayalso have a machine analysis algorithm incorporated such that a libraryof known environmental conditions may be updated each time audioprocessing module 3204 may be used.

For example, audio processing module 3204 may identify several voices,indicating that the apparatus is in a room with multiple people. In thisexample, image processing module 3201 may identify one or more chairsand/or a conference table, indicating that the apparatus is in aconference room.

Apparatus 110 may use the information from the audio and image data tocause the mobile communications device to alert the user, via audio ortext, that the apparatus 110 is in a conference room.

Apparatus location module 3202 may be configured to receive image dataand/or audio data determined by image processing module 3201 and audioprocessing module 3204. In some embodiments, apparatus location module3202 may receive location data from one or more apparatus components,e.g., a GPS unit, or from a mobile communications device associated withthe user of apparatus 110. Apparatus location module 3202 may determinelocation information indicating the location of the apparatus 110 basedon image information collected by image processing module 3201 and/oraudio processing module 3204. For example, objects detected in theenvironment of the apparatus 110 may indicate its location. Apparatus110 may use machine learning or other process described above torecognize a collection of one or more objects as being associated with aparticular room. In some embodiments, apparatus location module 3202 mayalso receive motion data from a motion sensor of apparatus 110 anddetermine the location of the location of the apparatus based on thecombination of motion, image, and/or audio data.

For example, upon receiving the command from the user to locate theapparatus 110, mobile communications device may communicate with, ortransmit a “ping” to apparatus 110. Apparatus location module 3202 mayconnect with mobile communications device and may transmit its locationdata, e.g., GPS coordinates or proximity to the mobile communicationsdevice. Location data may include, for example, GPS coordinates, anestimated distance from the mobile communications device, and/or heightfrom a reference point such as the floor. In some embodiments, apparatuslocation module 3202 may determine the distance from apparatus 110 tothe mobile communications device by analyzing GPS coordinates of theapparatus 110 in comparison to GPS coordinates of the mobilecommunications device.

In other embodiments, if the apparatus 110 is not in the immediatevicinity of the user, e.g., the apparatus is not in the same building,apparatus location module 3202 may access a map on a server and providedirections from the GPS location of the mobile communications device tothe GPS location of the apparatus 110. In this example, mobilecommunications device may provide turn-by-turn audio directions, displaya map indicating a route, or may transmit directions and route map toone or more of a GPS unit, a smart vehicle including a GPS unit, and/oranother device specified by the user.

Direction module 3203 may receive data from image processing module3201, audio processing module 3205, and apparatus location module 3202,determine a set of instructions to direct the user to the apparatus 110,and output those directions. The set may or may not be an optimal set ofdirections. Depending on the distance of the apparatus 110 from theuser, mobile communications device may either provide directions to alocation or may display and/or provide audio indicating the location ofapparatus 110. For example, if apparatus 110 is over 500 feet away frommobile communications device, direction module 3203 may output a listingof turn-by-turn directions to the location of the apparatus 110. Inanother example, if apparatus 110 is a few feet away, e.g., one or twofeet, from the mobile communications device, apparatus 110 may transmitinstructions to the mobile communications device to output audioindicating the location of the apparatus 110.

In one example, if the image sensor 220 is on a table, it may capturethe table surface and other furniture such as a sofa and armchair. Inthis example, apparatus location module 3202 may determine the locationof the apparatus 110, e.g., that the apparatus 110 is on a table in aliving room, and direction module 3203 may generate and transmitinstructions for the mobile communications device to provide audiooutput, “The device is on the table in the living room.” In otherembodiments, direction module 3203 may generate and image and/or iconindicating the location of apparatus 110. In another embodiment,direction module 3203 may transmit instructions to the user's mobilecommunications device to alert the user as he gets closer to themisplaced apparatus 110.

In some embodiments, the location information determined by directionmodule 3203 may be configured to cause the mobile communications deviceto provide an image representing the location of the wearable apparatus.In some embodiments, a user of the mobile communications device isdepicted in the at least one image, and the information includes aposition of the user relative to the wearable apparatus. The mobilecommunications device may be configured to audibly output the positionof the user relative to the wearable apparatus based on the informationabout the location of the wearable apparatus.

In some embodiments, the housing of image sensor 220 may be designedsuch that the image sensor 220 does not land face down, such that it canalways collect an image of its environment. For example, image sensor220 may have a curved housing such that, if the sensor falls face downon a table, it may still collect some image information. In anotherexample, the apparatus 110 may be configured to change the capturingangle of the image sensor 220, or another parameter, e.g., zoom orbrightness, to capture more images if image processing module 3201cannot extract enough data to locate the apparatus 110 from an initialset of images. In some embodiments, apparatus 110 may use feedback fromone or more image processing algorithms to determine areas of the fieldof view to capture or to determine an amount by which to alter aparameter, e.g., capturing angle.

Modules 3201-3204 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, they may be stored, for example, in memory 550.However, in some embodiments, any one or more of modules 3201-3204 may,for example, be stored in processor 540 and/or located on server 250,which may include one or more processing devices. Processing devices ofserver 250 may be configured to execute the instructions of modules3201-3204. In some embodiments, aspects of modules 3201-3204 may includesoftware, hardware, or firmware instructions (or a combination thereof)executable by one or more processors, alone or in various combinationswith each other. For example, modules 3201-3204 may be configured tointeract with each other and/or other modules of server 250 and/or awearable camera system and/or a mobile communications device to performfunctions consistent with disclosed embodiments. In some embodiments,any of the disclosed modules may each include dedicated sensors (e.g.,IR, image sensors, etc.) and/or dedicated application processing devicesto perform the functionality associated with each module.

FIG. 33A is an exemplary scenario in which a user 3301 is attempting tolocate apparatus 110, which has fallen underneath a sofa 3302. Toinitiate apparatus location, user 3301 may open a mobile application ona mobile communications device (not shown) and input informationindicating that he cannot locate the apparatus 110. After receiving acommunication, or “ping,” from the mobile communications device,apparatus 110 may execute instructions causing image sensor 220 tocollect one or more images of the apparatus' environment Imageprocessing module 3201 may process image data to identify one or moreobjects in the collected one or more images. Objects may include theedge of the sofa 3302, the floor, the feet of the user 3301, etc.Direction module 3203 may analyze these images to determine that theapparatus 110 is on the floor, under a piece of furniture, and to theright of the user 3301. In another example, audio data analyzed by audioprocessing module 3205 may indicate footsteps and/or the voice of theuser. Such audio data may enable the processor 210 to determine that theuser 3301 is in the vicinity of the apparatus 110 and may enable theprocessor 210 to determine the direction and distance of the user 3301from the apparatus 110.

Direction module 3203 may transmit instructions and/or locationinformation the mobile communications device. For example, directionmodule 3203 may wirelessly transmit one or more images collected byimage sensor 220 and instructions to cause the mobile communicationsdevice to display the one or more images, such that a user may also tryand find the device based on objects seen in the images. Additionally,direction module 3203 may transmit instructions for the mobilecommunications device to output spoken audio, for example, “The deviceis under the furniture to your right.”

FIG. 33B is an exemplary scenario in which the user 3301 is attemptingto locate apparatus 110, which is inside a briefcase 3303 on a desk3304. To initiate apparatus location, user 3301 may open a mobileapplication on a mobile communications device (not shown) and inputinformation indicating that he cannot locate the apparatus 110. Afterreceiving a communication, or “ping,” from the mobile communicationsdevice, apparatus 110 may execute instructions causing image sensor 220to collect one or more images of the apparatus' environment Imageprocessing module 3201 may process image data to identify one or moreobjects in the collected one or more images. Objects may include, theedges of the briefcase 3303, a portion of the desk chair 3305, and aportion of user 3301. Audio information may include the voice of theuser. For example, audio processing module 3204 may determine how closethe user 3301 is to apparatus 110 based on the volume of the user'svoice. Direction module 3203 may analyze these images to determine thatthe apparatus 110 is on the desk inside a container, e.g., the briefcase3303 and on an elevated surface. Direction module 3203 may determinethat apparatus 110 is on a surface based on the detected objects. Forexample, if image sensor 220 collects images of the top of a chair andthe midsection of the user, direction module 3203 may determine that theapparatus 110 is on a table, or other elevated surface.

Direction module 3203 may transmit instructions and/or locationinformation the mobile communications device. For example, directionmodule 3203 may wirelessly transmit one or more images collected byimage sensor 220 and instructions to cause the mobile communicationsdevice to display the one or more images. Additionally, direction module3203 may transmit instructions for the mobile communications device tooutput spoken audio, for example, “The device is in a container on atable to your left.” In some embodiments, mobile communications devicemay display icons indicating the relative positions and proximity of theapparatus 110 and user 3301.

FIG. 33C is an exemplary scenario in which the user 3301 is attemptingto locate apparatus 110, which is inside the user's pocket 3306. Toinitiate apparatus location, user 3301 may open a mobile application ona mobile communications device (not shown) and input informationindicating that he cannot locate the apparatus 110. After receiving acommunication, or “ping,” from the mobile communications device,apparatus 110 may execute instructions causing image sensor 220 tocollect one or more images of the apparatus' environment. However, inthe user's pocket 3306, image sensor 220 may be unable to collect imagessince the image sensor 220 may be blocked by fabric. Image processingmodule 3201 may communicate with direction module 3203 to indicate thatthe apparatus 110 is inside a container, e.g., a pocket, and cannotprovide image data. In this example, direction module 3203 may receivelocation information, such as proximity to the mobile communicationsdevice, from apparatus location module 3202. If the location of theapparatus 110 is close, e.g., one foot, to the location of the mobilecommunications device, direction module 3203 may instruct the mobilecommunications device to display or output audio instructing the userthat apparatus 110 is in a pocket. In another embodiment, directionmodule 3203 may execute instructions causing the apparatus 110 to emit asound to audibly guide the user to the apparatus.

In some embodiments, apparatus 110 may include a motion sensor. Forexample, if user 3301 is walking and cannot locate apparatus 110, whichis in his pocket 3306, the motion sensor may collect motion dataindicative of a gait pattern. Apparatus location module 3202 may use themotion data in combination with image and/or audio data to determinethat the apparatus 110 is in the user's pocket 3306. In someembodiments, apparatus 110 may be a gait recognition algorithm todetermine if the gait pattern detected by the motion sensor matches theuser's gait pattern.

FIG. 34 is a flowchart of a method 3400 for locating an apparatus anddisplaying location information on a mobile communications device.Method 3400 may be implemented by a general-purpose computer or aspecial-purpose computer built according to embodiments of the presentdisclosure. For example, method 3400 may be executed by at least one ofprocessors 210, 210 a, 210 b, and 540. In some embodiments, the steps ofmethod 3400 may be performed by wearable apparatus 110. In otherembodiments, the steps of method 3400 may be performed by wearableapparatus 110 and one or more external devices (e.g., a processorincluded in an external server that receives data from wearableapparatus 110 over a network and/or a processor included in an externaldevice such as a laptop, a smartwatch, a smartphone, a tablet, or earphones, etc.).

At step 3401, the processor may receive, from a mobile communicationsdevice wirelessly paired with the wearable apparatus, a command toprovide location information of the wearable apparatus. In someembodiments, the processor may establish a wireless pairing between thewearable apparatus and the mobile communications device. If theapparatus is misplaced, the processor may receive user input for pingingthe wearable apparatus. In response to the user input, the processor maywirelessly transmit, from the mobile communications device, a ping tothe wearable apparatus including an instruction for the wearableapparatus to capture at least one image of the surroundings of thewearable apparatus.

At step 3402, the processor may analyze, in response to the command, atleast one image depicting an environment of the wearable apparatus todetermine a location of the wearable apparatus. For example, directionmodule 3203 may receive image information from image processing module3201 and identify the location of the apparatus 110 based on the imageinformation. In other embodiments, direction module 3203 mayadditionally receive location data from apparatus location module 3202and may use the location data in concert with the image information todetermine the location of the apparatus.

At step 3402, in some embodiments, the processor may also analyze audioinformation in combination with image information to determine alocation of the wearable apparatus. For example, direction module 3203may receive image information from image processing module 3201 andaudio information from audio processing module 3204, and identify thelocation of the apparatus 110 based on the combination of image andaudio information. In other embodiments, direction module 3203 mayadditionally receive location data from apparatus location module 3202and may use the location data in concert with the image and/or audioinformation to determine the location of the apparatus. One of ordinaryskill will recognize that, in some embodiments, the processor mayanalyze any combination of images and/or audio data to determine alocation of the wearable apparatus.

At step 3403, the processor may cause information about the location ofthe wearable apparatus to be transmitted to the mobile communicationsdevice. As previously described, information about the location of theapparatus 110 may include GPS coordinates, images collected by the imagesensor 220, and/or directions, location of the apparatus 110 relative tothe user. The processor may additionally transmit executableinstructions to cause the mobile communications device to display and/oraudibly output information about the location of the apparatus 110.

Wearable Apparatus Providing Feedback to Adjust a Field of View

In some embodiments, wearable apparatus 110 may collect informationrelated to the environment of the user of the wearable apparatus 110.Furthermore, in some instances, image sensor 220 of apparatus 110 may bepositioned in such a way that it is obstructed from capturing a completefield of view. In other instances, the image sensor 220 may be orientedin such a way that it is unable to capture the complete field of viewand/or portions of the field of view containing information about theuser's environment. In some embodiments, the apparatus 110 may beconfigured to detect an obstruction or poor orientation of the imagesensor 220 and communicate notification or correction information to theuser. For example, apparatus 110 may suggest to the user to repositionor re-orient the image sensor 220.

Currently, wearable devices or apparatuses may include image sensingcomponents.

However, the user may be required to ensure the image sensing componentis clear of obstruction and/or capable of detecting a required field ofview. In some circumstances, the image sensing component may beobstructed unbeknownst to the user, thereby rendering the image sensingcomponent unable to capture images t usable for providing meaningfulfeedback.

As described above, in some embodiments, any one of apparatus 110 orcomputing device 120, via processor 210 or 540, may further processcaptured image data and determine that all or a portion of the field ofview of image sensor 220 is obstructed. In some embodiments, apparatus110 may analyze at least one image captured by the wearable image sensorto determine a positioning issue related to the wearable image sensor220. Responsive to the determination of a positioning issue, apparatus110 may determine correction information for resolving the positioningissue and cause the correction information to be provided to the user ofapparatus 110.

A positioning issue may arise when image sensor 220 is fully orpartially blocked and thus cannot collect image data. In someembodiments, the positioning issue may include an obstruction of thewearable image sensor 220 and/or an impediment to the quality of imagecapture. In other embodiments, the positioning issue may include anobstruction of the wearable image sensor 220 by the user of apparatus110. For example, apparatus 110 may be worn by the user in such a mannerthat image sensor 220 is blocked by the user or by a piece of the user'sclothing. In another example, image sensor 220 may be positioned behinda gauzy, or translucent, article of clothing, e.g., a linen scarf. Thepositioning issue may be the resulting impediment to image qualitycaused by the scarf. In other embodiments, the positioning issue mayinclude an orientation of the wearable image sensor 220. For example, ifthe image sensor 220 is mounted on glasses, e.g., glasses 130, the usermay remove the glasses and place them on top of his head such that imagesensor 220 is oriented toward the sky or ceiling. In this example, imagesensor 220 may be unable to capture image information relevant to theuser, i.e., information about the user's environment.

In response to the detection of a positioning issue of the image sensor220, apparatus 110 may provide the user with correction information. Insome embodiments, correction information may include a recommendation tomove the wearable apparatus. For example, if the image sensor 220 isblocked by the user's scarf, correction information may be to wear theapparatus 110 at a different location on the user's body. In someembodiments, correction information may include a recommendation toremove or reposition the obstruction. In the above example, correctioninformation may include a recommendation to the user to remove and/orreposition the scarf. In embodiments in which the positioning issue isrelated to the orientation of the image sensor 220, correctioninformation may include a recommendation to change the orientation ofthe wearable image sensor 220.

In some embodiments, correction information is provided to a mobilecommunications device wirelessly paired with the wearable apparatus. Insome embodiments, the correction information is provided as text output.For example, the correction information may be configured to cause themobile communications device to output the correction information astext. For example, correction information may be displayed on a screenof the mobile communications device via a mobile application. In anotherembodiment, the correction information is configured to cause the mobilecommunications device to audibly output the correction information, viaa speaker of the mobile communications device.

In some embodiments, correction information may include a genericrecommendation to adjust the wearable apparatus. For example, correctioninformation may cause the user's mobile communications device to alertthe user, via a text and/or audio message, that the image sensor isunable to collect sufficient image data. An exemplary message may be,for example, “Image sensor is blocked. Please adjust device.”

In another embodiment, correction information may include executableinstructions configured to cause the at least one processor toautomatically resolve the positioning issue. For example, thepositioning issue may be the orientation of the image sensor 220.Correction information may include instructions such that the processoradjusts the orientation and/or angle of the image sensor within theapparatus so the sensor can capture a greater portion of the field ofview.

FIG. 35 illustrates an exemplary embodiment of a memory 3500 containingsoftware modules consistent with the present disclosure. Included inmemory 550 are an image processing module 3501, a positioning module3502, and a correction module 3503. Modules 3501, 3502, 3503 may containsoftware instructions for execution by at least one processing device,e.g., processor 210, included with a wearable apparatus. Imageprocessing module 3501, positioning module 3502, and correction module3503 may cooperate to process an image from an image sensor, e.g., imagesensor 220, identify a positioning issue, determine correctioninformation based on the positioning issue, and output the correctioninformation to the user. Memory 3500 may be separate from and/orintegrated with memory 550 of FIG. 6, described above Similarly,orientation identification module 601, orientation adjustment module602, and motion tracking module 603 of FIG. 6 may operate in tandem orconcurrently with image processing module 3501, positioning module 3502,and correction module 3503 of FIG. 35.

Image processing module 3501 may be configured to identify one or moreobjects and/or lighting levels from one or more images. In someembodiments, objects in the one or more images may be identified throughthe application of one or more image classification techniques. Forexample, at least one image classification technique may be used toclassify at least one feature of an image. In some embodiments, an imageclassification technique may include one or more of image enhancement,edge detection, image analysis, and data extraction. Specific examplesof the methods for identifying at least one object in an image areexemplary only, and a person of ordinary skill in the art will recognizeother methods for identifying the at least one positioning issue thatremains consistent with the present disclosure. In some examples, atleast one object may be detected using an object detection algorithm,using a neural network trained to detect objects and/or objects inimages, using a machine learning system, using a pattern recognitionsystem, and so forth. In some embodiments, image processing module 3501may identify a lighting level of the environment of the user.

Positioning module 3502 may be configured to receive image data fromimage processing module 3501 and identify the positioning issue. Apositioning issue may be a blockage of the image sensor 220 ororientation of the image sensor 220 such that it is unable to capturerelevant image information of the environment of the user. For example,image processing module 3501 may analyze one or more captured images todetermine that the image sensor 220 is not collecting image data from aportion of the field of view and positioning module 3502 may associatethe lack of data with a partial blockage of the image sensor 220. Inthis example, positioning module 3502 may determine which side of theimage sensor 220 is blocked, e.g., relative to the user, an object isblocking the left portion of the sensor. In another example, imageprocessing module 3501 may not receive any image data, indicating thatthe image sensor 220 is completely blocked. In another embodiment, ifthe positioning issue is the orientation of the image sensor 220, imageprocessing module 3501 may identify a flat surface, e.g., a ceiling ofthe room that the user is in, or the sky. In this example, positioningmodule 3502 may determine based on information from image processingmodule 3501, that the positioning issue is that the image sensor isoriented upwards. Positioning module 3502 may also have a machineanalysis algorithm incorporated such that a library of known blockagesand/or orientations may be updated each time image processing module3501 and positioning module 3502 may be used.

Correction module 3503 may receive positioning issue information frompositioning module 3502 and determine correction information including arecommendation to the user to resolve the positioning issue.Recommendations may include instructions to move, reposition, and/orre-orient the image sensor 220. In other embodiments, a recommendationmay be that the user repositions himself and/or an article of clothing.Correction module 3503 may be configured to transmit the correctioninformation as text and/or audio instructions to a mobile communicationsdevice wirelessly paired to apparatus 110. The apparatus 110 maytransmit the correction information by communicating with a device, viaany known wireless standard (e.g., wifi, Bluetooth®, etc.), as well asnear-field capacitive coupling, and other short range wirelesstechniques, or via a wired connection.

In some embodiments, correction module 3503 may receive data indicativeof a positioning issue arising from the orientation of the image sensor220. Correction module 3503 may communicate with one or more orientationmodules to automatically modify the orientation of the image sensor tocorrect the positioning issue identified by positioning module 3502.

Modules 3501-3503 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, they may be stored, for example, in memory 550.However, in some embodiments, any one or more of modules 3501-3503 may,for example, be stored in processor 540 and/or located on server 250,which may include one or more processing devices. Processing devices ofserver 250 may be configured to execute the instructions of modules3501-3503. In some embodiments, aspects of modules 3501-3503 may includesoftware, hardware, or firmware instructions (or a combination thereof)executable by one or more processors, alone or in various combinationswith each other. For example, modules 3501-3503 may be configured tointeract with each other and/or other modules of server 250 and/or awearable camera system to perform functions consistent with disclosedembodiments. In some embodiments, any of the disclosed modules may eachinclude dedicated sensors (e.g., IR, image sensors, etc.) and/ordedicated application processing devices to perform the functionalityassociated with each module.

FIG. 36A is an exemplary scenario in which the system may identify apositioning issue in which the image sensor of the apparatus 110 ispartially blocked by the jacket 3601 of the user 3602. Apparatus 110 maybe worn by the user 3602 on a belt 3603. Belt 3603 may be the same as orsimilar to belt 150. If the user 3602 is wearing a jacket 3601, theapparatus 110 may become partially obstructed. Images captured by imagesensor 220 may lack data corresponding to the obstructed portion of thefield of view. Positioning module 3502 may receive this imageinformation from image processing module 3501 and determine that apositioning issue is that the left side (corresponding to the user'sleft side) of the image sensor is obstructed. Correction module 3503 maydetermine instructions such as, for example, “Clothing is blocking thedevice. Please reposition device or clothing.” These instructions may betransmitted wirelessly to the user's mobile communications device anddisplayed on the screen of the mobile communications device and/oraudibly output from a speaker of the user's mobile communicationsdevice.

FIG. 36B is an exemplary scenario in which the system may identify apositioning issue in which the image sensor of the apparatus 110 ispartially blocked by the user 3604. Apparatus 110 may be worn by theuser 3604 on a necklace 3605. Necklace 3605 may be the same as orsimilar to necklace 140. While wearing the device, the user 3604 mayblock the apparatus 110, for example, by crossing her arms. Imageprocessing module 3501 may analyze one or more images captured by imagesensor 220 to determine that the lower portion of the field of view isblocked. Image processing module 3501 may use one or more techniques, aspreviously described, to characterize objects in the environment of theuser, including the object causing the obstruction. Image processingmodule 3501 may analyze the geometry and/or curvature of the blockage todetermine that the blockage is caused by a human body part, e.g., anarm. Positioning module 3502 may receive this information and determinethat the lower portion of the field of view is blocked by the user'sarm.

In this example, correction information determined by correction module3503 may be to recommend the user 3604 move her arm. In someembodiments, for example, if correction information is output to theuser 3604 who moves her arm, and then the user 3604 assumes a positionat a later time such that her arm is again blocking the sensor,correction module 3503 may analyze the sequence of blockages andrecommend that the user 3604 move the apparatus 110 to avoid subsequentobstructions. As previously described, correction module 3503 maytransmit correction instructions to the user's mobile communicationsdevice and display the instructions on the screen of the mobilecommunications device.

FIG. 36C is an exemplary scenario in which the system may identify apositioning issue in which the image sensor of the apparatus 110 isoriented such that the image sensor 220 cannot collect relevant imageinformation. Apparatus 110 may be worn by the user 3606 on a watch 3607.While wearing the watch, the user 3606 may engage in an activity, suchas typing on a keyboard or writing, that causes the apparatus 110 to beoriented upwards. Image processing module 3501 may analyze one or moreimages captured by image sensor 220 and identify, for example, the chinof the user 3606 and/or the ceiling of the room the user 3606 is in.Image processing module 3501 may use one or more techniques, aspreviously described, to characterize objects in the environment of theuser 3606. Positioning module 3502 may receive this information, e.g.,the identification of the chin of the user 3606 and determine that theapparatus 110 is oriented upward.

In this example, correction information determined by correction module3503 may be to recommend the user 3606 move his arm or to move theapparatus 110 such that the field of view captured by image sensor 220is consistent with the field of view seen by the user 3606. In someexamples, such correction information may recommend that the user removethe apparatus (i.e., no longer wear it as watch), and position theapparatus such that the user may continue an activity (e.g., typing)while the field of view of image sensor 220 may monitor a particularregion (e.g., a display). As previously described, correction module3503 may transmit correction instructions to the user's mobilecommunications device and display the instructions on the screen of themobile communications device.

FIG. 37 is a flowchart of a method 3700 for determining a positioningissue related to a wearable image sensor and providing correctioninformation, based on the positioning issue, to the user according to adisclosed embodiment. Method 3700 may be implemented by ageneral-purpose computer or a special-purpose computer built accordingto embodiments of the present disclosure. For example, method 3700 maybe executed by at least one of processors 210, 210 a, 210 b, and 540. Insome embodiments, the steps of method 3700 may be performed by wearableapparatus 110. In other embodiments, the steps of method 3700 may beperformed by wearable apparatus 110 and one or more external devices(e.g., a processor included in an external server that receives datafrom wearable apparatus 110 over a network and/or a processor includedin an external device such as a laptop, smartwatch, smartphone, tablet,or ear phones, etc.).

At step 3701, the processor may analyze at least one image from awearable image sensor, e.g., sensor 220. For example, image sensor 220may capture one or more images of the environment of the user. Imageprocessing module 3501 may analyze the one or more images to identifyone or more objects, a lighting level, and/or a captured field of view.In some embodiments, processor 210 may receive a plurality of imagesfrom the wearable image sensor and perform image processing on at leastsome of the images.

At step 3702, the processor may determine a positioning issue based onthe analysis of the at least one image. In some embodiments, processor210 may determine, based on the image processing, an impediment toquality image capture. As previously described, positioning module 3502may receive image information from image processing module 3501 andassociate the image information with a positioning issue, such as ablockage, impediment, or inefficient orientation of the image sensor220.

At step 3703, the processor may determine correction information forresolving the positioning issue. In some embodiments, the processor maydetermine, in response to the determined impediment, correctioninformation for resolving the positioning issue. For example, correctionmodule 3503 may receive information indicating a positioning issue frompositioning module 3502 and determine correction information to resolvethe positioning issue. As previously described, the correctioninformation may include a recommendation to move or reposition apparatus110 to resolve the positioning issue.

In some embodiments, correction module 3503 may receive information frompositioning module 3202 indicative of a positioning issue, but thepositioning information may not indicate a cause of the positioningissue. If a specific cause is not determined by positioning module 3502,correction module 3503 may provide a general message. For example, ifthe image sensor does not collect any images including extractable data,e.g., if the sensor is completely blocked, positioning module 3502 maybe unable to determine the cause of the blockage. In this example,correction module 3503 may cause the mobile communications device tooutput a generic message such as, “Image sensor is blocked.”

At step 3704, the processor may cause the correction information to beprovided to a user of the wearable apparatus. Providing the correctioninformation may include transmitting the correction information to amobile communications device wirelessly paired with the apparatus 110.Correction information may include textual, visual, and/or audioinstructions for the user to resolve the positioning issue. Theprocessor may further cause instructions to be transmitted to the mobilecommunications device to display the text instructions or to outputaudio instructions using text-to-speech software. In some embodiments,the processor may present the correction information on a display of thepairable mobile communications device.

In some embodiments, method 3700 may include optional step 3705. Thepositioning issue identified by the positioning module 3502 may notrequire the user to re-position or re-orient the image sensor 220. Ifthe positioning issue may be resolved by the apparatus 110 itself, atstep 3705, the processor may cause the apparatus 110 and/or image sensor220 to automatically correct the positioning issue. For example,automatically correcting the positioning issue may include modifying oneor more of the angle of the sensor, sensitivity of the sensor, orresolution of the images captured.

In one example, a user may wear the apparatus on an article of clothingthat is oriented at a slight angle relative to the body of the user.Positioning module 3502 may determine, based on image data, that imagesensor 220 is not capturing the full field of view from the same angleas the user. In this example, the processor may resolve the positioningissue by automatically rotating the image sensor 220 such that thesensor is facing away from the user in parallel with the user's line ofsight.

Wearable Apparatus for Name Tagging

In some embodiments, the disclosed apparatus and methods may facilitatethe association of captured facial images with names by employingnatural language processing to extract a name from a conversation, whichmay be associated with the facial image. To ensure accuracy, the usermay be queried for confirmation before the name is stored. The disclosedapparatus and methods may further look-up an identity based on asubsequent facial image, after a subsequent encounter.

As shown in FIG. 38, in combination with other modules discussed above,association of facial images with names may be partitioned into, forexample, four modules, including a facial image association module 3802,an identity lookup module 3804, a sound processing module 3806, and adisplay module 3808. In some embodiments, modules 3802, 3804, 3806, and3808 may be stored in memory unit 550 of apparatus 110 and/or 120, ormodules 3802, 3804, 3806, and 3808 (or portions thereof) may be storedremotely from apparatus 110 (e.g., stored in a server accessible toapparatus 110 via, for example, wireless transceiver 530).

Facial image association module 3802, which may store instructionsexecutable by processor 210, may handle various tasks relating toassociating text generated based on a spoken name with a facial image.Such tasks may rely upon input from various sensors and sensing systemsassociated with the apparatus 110, as described above.

Facial image association module 3802 may receive, from a wearable imagesensor (such as image sensor 220), a facial image of an individual withwhom a user of the wearable apparatus is interacting. The facial imagemay be in any suitable still or motion imaging format, and theindividual depicted in the facial image may comprise a human person. Inan embodiment, the individual depicted in the image may comprise arepresentation of a person, a rendered, animated, or illustrateddepiction of a person, or any other visual subject that may appear in anoutput of the wearable image sensor.

Facial image association module 3802 may store, in a memory (e.g.,memory unit 550), text associated with the individual in a mannerassociating the text with the facial image. Any form of data structure,file system, or algorithm for associating the text with the facial imagemay be employed. The text may comprise audio captured during aninteraction with the individual. For example, the text may comprise aspoken name of the individual captured in a conversation with theindividual and/or related to the individual. In some embodiments, facialimage association module 3802 may receive text from sound processingmodule 3806 (discussed further below). While the description thatfollows focuses on text related to a name of the individual, the textmay comprise any piece of information associated with the individual,for example, a title, common interest, relationship with one or moreother individuals, etc.

In some embodiments, prior to storing the text associated with theindividual, facial image association module 3802 may prompt the user toconfirm a presumed name of the individual. For example, facial imageassociation module 3802 may cause a display to the user of a presumedname of the individual. The display may be carried out at least in partby display module 3808 (described in further detail below). In someembodiments, prior to storing the text associated with the individual,facial image association module 3802 may enable the user to alter thetext. Any known means of altering text may be employed. For example,image association module 3802 may prompt display module 3808 to displaya cursor to enable editing characters of the text. Additionally oralternatively, image association module 3802 may prompt display module3808 to display proposed replacement candidates for the text.

In some embodiments, facial association module 3802 may store textassociated with an individual in association with a different image thanan image received by the wearable image sensor at or near the time thatthe audio was captured during the interaction with the individual. Thatis, facial association module 3802 may store the text associated withthe spoken name in a manner associating the text with the facial imageby storing the text in association with a previously captured facialimage.

In some embodiments, facial association module 3802 may receive textfrom sources other than captured audio and associate the text with afacial image of the individual. For example, facial association module3802 may be programmed to determine a name based on analysis of text inan image. Facial association module 3802 may determine a name based on aname tag, identification card, etc., which appears in the image. Anysuitable process for analyzing the text may be employed in suchanalysis, such as known optical character recognition (OCR) techniques.

Identity lookup module 3804 may store instructions executable byprocessor 210 and may handle various tasks relating to looking up anidentity of an individual. For example, identity lookup module 3804 mayreceive, from the wearable image sensor, a facial image of theindividual. The received facial image may be a subsequent facial image.That is, the facial image may be received based on an encounter with theindividual subsequent to an encounter for which text associated with theindividual was stored by facial image association module 3802 in amanner associating the text with a facial image.

Identity lookup module 3804 may perform a look-up of an identity of anindividual based on a facial image, such as a subsequent facial image.The look-up may be performed based on any known image matching or facialidentification process, or any combination of suitable technologies.Responsive to performing the lookup, identity lookup module 3804 mayreceive, from a memory (e.g., memory unit 550 of apparatus 110 and/or120), text of the spoken name of the individual.

Sound processing module 3806 may store instructions executable byprocessor 210 and may handle various tasks relating to reception andmanagement of audio information. Sound processing module 3806 mayreceive sound data captured during an interaction with an individual.Sound processing module 3806 may obtain the sound data from a soundsensor in a housing that includes the wearable image sensor, such asthat of apparatus 120. Additionally or alternatively, in someembodiments, the sound data may be obtained from a sound sensor includedwithin a computing device 120 in communication with apparatus 110, asdescribed above. In such embodiments, the sound data may be obtainedfrom a mobile communications device such as a smartphone or tablet.

Sound processing module 3806 may process at least a portion of receivedsound data to determine a spoken name of the individual. Processing mayinclude converting the captured sound data into an appropriate formatfor storage or further processing. In some embodiments, sound processingmodule 3806 may convert the spoken name to text, using any knownspeech-to-text process or technology. Sound processing module 3806 mayalso access a database of names and compare the names with the at leasta portion of received sound data. More specifically, words parsed fromthe sound data may be compared with names in the database. The databaseof names may be stored in memory unit 550 of apparatus 110 and/or 120,and additionally or alternatively may be included in a remote serveraccessible over one or more networks. Sound processing module 3806 mayprocess the portion of the sound data to determine the spoken name ofthe individual at least in part by accessing the remote server.

Display module 3808 may store instructions executable by processor 210and may handle various tasks relating to display of identity informationrelated to an individual. For example, display module 3808 may cause adisplay in text of the name of the individual on a display associatedwith apparatus 110. In some embodiments, display module 3808 may causedisplay of text on a display integrated into apparatus 110. Any displaytechnology known in the art and suitable for inclusion in apparatus 110may be used. In some embodiments, display module 3808 may cause displayof text on one or more of a liquid crystal display (LCD), light-emittingdiode (LED) display, organic light-emitting diode (OLED) display, etc.Additionally or alternatively, display module 3808 may cause a displayof the name of the individual on a device (e.g., computing device 120)paired with, networked with, or otherwise in communication with thewearable apparatus, such as a mobile communications device.

FIG. 39A is an example of a flowchart 3900 representing an exampleprocess for association of captured facial images with names, byemploying natural language processing to extract a name from theconversation. At step 3902, at least one processing device (e.g.,processor 210) of a wearable apparatus may receive, from a wearableimage sensor, a facial image of an individual with whom a user of thewearable apparatus is interacting. In some embodiments, the facial imagemay be received by facial image association module 3802, as describedabove.

At step 3904, the at least one processing device may receive sound datacaptured during the interacting. The sound data may be obtained from asound sensor in a housing that includes the wearable image sensor. Forexample, as described above, the sound data may be received by soundprocessing module 3806. As further described above, the sound data maybe obtained from a sound sensor included within a mobile communicationsdevice.

At step 3906, the at least one processing device may process at least aportion of the sound data to determine a spoken name of the individual.The processing may be performed by sound processing module 3806, asdescribed above. As further described above, processing at least aportion of the sound data at step 3906 may include accessing a databaseof names and comparing words parsed from the sound data with names inthe database, wherein the database may be included in a remote server.

At step 3908, the at least one processing device may convert the spokenname to text. The conversion may be performed by sound processing module3806, as described above.

At step 3910, the at least one processing device may store, in memory,text associated with the spoken name in a manner associating the textwith the facial image. The storing may be performed by facial imageassociation module 3802, as described above. In some embodiments, thestoring at step 3910 may include storing the text in association with apreviously captured facial image.

At step 3912, the at least one processing device may receive, from thewearable image sensor, a subsequent facial image of the individual.Receiving the subsequent facial image may be performed by identitylookup module 3804, as described above. The subsequent facial image maybe received at step 3912, for example, after a subsequent encounter withthe individual.

At step 3914, the at least one processing device may perform a look-upof an identity of the individual based on the subsequent facial image.The look-up may be performed by identify lookup module 3804, asdescribed above.

At step 3916, the at least one processing device may receive, from thememory, the text of the spoken name of the individual. The text of thespoken name may be received by identity lookup module 3804, as describedabove.

At step 3918, the at least one processing device may cause a display intext of the name of the individual on a mobile communications devicepaired with the wearable apparatus. The display of the name of theindividual may be performed by display module 3808, as described above.

In some embodiments, the at least one processing device may encounterambiguity or uncertainty in associating captured facial images withnames. For example, environmental noise may result in occasional speechto text inaccuracy, or a spoken name may be subject to more than onespelling. To resolve such issues, FIG. 39B is an example of additionalsteps of flowchart 3900, representing an example process for confirmingthe accuracy of the text of a name, before and/or as a part of storingthe name in association with a facial image.

At step 3909 a the at least one processing device may cause a display tothe user of a presumed name of the individual. Step 3909 a may beperformed by facial image association module 3802, as described above.Step 3909 a may commence after converting the spoken name to text atstep 3908, but may take place prior to storing the text in associationwith the facial image at step 3910. At step 3909 a, the at least oneprocessing device may prompt the user to confirm the displayed presumedname of the individual.

FIG. 40 depicts an example of an interface 4002, displayed on device4006, which may be configured consistent with step 3909 a. The interfacemay be displayed on a device such as apparatus 110 or computing device120, as described above in relation to display module 3808. As shown inFIG. 40, interface 4002 may include presumed name 4008, “John Smith.”Interface 4002 may also include a message 4010, prompting the user toconfirm presumed name 4008.

Returning to FIG. 39B, at step 3909 b the at least one processing devicemay receive user input related to accuracy of the presumed name. Step3909 a may be performed by facial image association module 3802, asdescribed above. Any means of confirmation may be used, such as touchscreen input, physical buttons, gestures received by an image sensor oraccelerometer, etc. In the example of FIG. 40, interface 4002 depictsinterface elements 4012 and 4014, which respectively representconfirmation and rejection of presumed name 4008. Other configurationsof confirming or rejecting the presumed name will be apparent to thoseof skill in the art. For example, in some embodiments, interface element4014 may be omitted, and a user may be enabled to edit presumed name4008 by selecting presumed name 4008 itself. If the input received atstep 3909 b indicates that the presumed name is accurate (step 3909 c;YES), flowchart 3900 may proceed to 3910, described above. If, however,the input received at step 3909 b indicates that the presumed name isnot accurate (step 3909 c; NO), flowchart 3900 may proceed to step 3909d.

At step 3909 d, the at least one processing device may receive userinput altering the text of the spoken name, thereby enabling the user toalter the text prior to storing at step 3910. Step 3909 a may beperformed by facial image association module 3802, as described above.Returning to the example of

FIG. 40, interface 4004 may be configured consistent with step 3909 d.Interface 4004 may be configured to provide functionality to allow theuser to modify the text of the spoken name. For example, as the names“John” and “Jon” may be pronounced identically, the user may wish tocorrect the spelling to accurately reflect the name of the individual.Interface 4006 may include a text box 4018 and a cursor 4020, tofacilitate such revision. Interface 4006 may also include a message4016, prompting the user to revise the name shown in text box 4018.After revision of the name, the user may select interface element 4022to confirm, and flowchart 3900 may proceed to step 3910, describedabove.

In some embodiments, since it may not be known apriori whether the useris meeting an individual for the first time or the individual's name isalready stored, after receiving the individual's facial image at step3902, flowchart 3900 may proceed to step 3914, and the at least oneprocessing device may look-up the identity of the individual. If anindividual is identified based on the look-up, flowchart 3900 mayproceed to step 3916, while otherwise flowchart 3900 may proceed to step3904.

In some embodiments, if an individual is identified based on the lookupat step 3914, the at least one processing device may receive inputsimilar to the input received at step 3909 b. The input may representconfirmation or rejection of the identity of the identified individualand/or may allow the user to edit the existing text of the name of theindividual identified in the lookup. Furthermore, either automaticallyor in response to input received from the user, the at least oneprocessing device may associate the received facial image with theidentity (e.g., the name) of the individual looked-up at step 3914.

Wearable Apparatus for Prioritizing Tasks

In some embodiments, the disclosed apparatus and methods may facilitateprioritizing tasks based on information identified from captured images.For example, feedback may be provided to a user to alert the user tochanges in task priority.

As shown in FIG. 41, in combination with other modules discussed above,prioritization of tasks may be partitioned into, for example, threemodules, including an image analysis module 4102, task prioritizationmodule 4104, and a feedback module 4106. In some embodiments, modules4102, 4104, and 4106 may be stored in memory unit 550 of apparatus 110and/or 120, or modules 4102, 4104, and 4106 (or portions thereof) may bestored remotely from apparatus 110 (e.g., stored in a server accessibleto apparatus 110 via, for example, wireless transceiver 530). Further,memory 550 may be configured to store a plurality of tasks.

As described herein, tasks may include any processes or objectives thatmay be accomplished by or for a user associated with apparatus 110.Tasks may include personal tasks such as appointments, schedule items,chores, shopping lists, exercise goals, health and wellness tasks, etc.

Additionally or alternatively, tasks may include tasks involved insatisfying requirements associated with a user's employment. Tasks maybe stored in a memory unit configured to store a plurality of tasks, forexample, memory unit 550 of apparatus 110 and/or 120, and/or tasks maybe stored remotely from apparatus 110 (e.g., stored in a serveraccessible to apparatus 110 via, for example, wireless transceiver 530).

Image analysis module 4102, which may store instructions executable byprocessor 210, may handle various operations relating to obtaininginformation related to a plurality of tasks. Such operations may relyupon input from various sensors and sensing systems associated with theapparatus 110, as described above. For example, image analysis module4102 may receive images from a wearable image sensor (such as imagesensor 220). The wearable image sensor may be configured to capture aplurality of images from an environment of a user of the wearableapparatus. The images may be in any suitable still or motion imagingformat, and the subject(s) depicted in the images may comprise anythingwhich may be interpreted to affect priority. In an embodiment, thesubject(s) depicted in the image may comprise a representation of aperson or object, a rendered, animated, or illustrated depiction of aperson or object, or any other visual subject that may appear in anoutput of the wearable image sensor.

In some embodiments, image analysis module 4102 may additionally oralternatively receive information from sources other than a plurality ofimages. For example, sound data may be captured and analyzed by imageanalysis module 4102 and/or another module of apparatus 110. Imageanalysis module 4102 may obtain sound data from a sound sensor in ahousing that includes the wearable image sensor, such as that ofapparatus 110. Additionally or alternatively, in some embodiments, sounddata may be obtained from a sound sensor included within a computingdevice 120 in communication with apparatus 110, as described above. Insuch embodiments, the sound data may be obtained from a mobilecommunications device such as a smartphone or tablet. Analysis mayinclude converting captured sound data into an appropriate format forstorage or further processing. In some embodiments, sound processingmodule 3806 may convert sound to text, using any known speech-to-textprocess or technology.

Image analysis module 4102 may analyze a plurality of images to obtaininformation related to at least some of a plurality of tasks.Information may be in any appropriate format for use by apparatus 110.Analysis may be performed by any appropriate method or algorithm. Forexample, comparison with known images may be performed, known traits maybe identified from within one or more of the images, optical characterrecognition may be performed to extract text from an image, etc.

The information related to at least some of the plurality of tasks maycomprise an information may be related to an estimated time to completeone or more of the plurality of tasks. In some embodiments, imageanalysis module 4102 may identify objects in one or more of the imagesindicative of the expected time to complete the task. For example, imageanalysis module 4102 may determine that the one or more images indicatethat the user is waiting in a line, driving in traffic, or walking in acrowd. The presence of objects associated with such conditions, such asother people, vehicles, etc. may be determined by image analysis module4102.

In some embodiments, the information related to at least some of theplurality of tasks may comprise information related to a personassociated with one or more of the plurality of tasks. For example,image analysis module 4102 may determine an identity of the person, aposition of a person, an activity a person is engaged in, whether aperson is newly present or absent, etc. As another example, in someembodiments, the information related to at least some of the pluralityof tasks may comprise a physical, mental and/or emotional state of aperson such as an acute medical problem, an estimated happiness level,patience level, stress level, anger level, etc. Such states may bedetermined from any appropriate detectable trait of the person, such asby analyzing body or facial features, posture, movement(s), gestures,etc.

In some embodiments, image analysis module 4102 may analyze a pluralityof images to obtain information related to a location associated withone or more of a plurality of tasks. For example, image analysis module4102 may identify text or other identifying information associated withone or more roads, cities, states, other geographic areas. In someembodiments, image analysis module 4102 may identify one or moresymbols, landmarks, buildings, organizations, businesses, etc. known tobe associated with a particular location or region. Image analysismodule 4102 may analyze a plurality of images to obtain informationrelated to a hazardous situation, such as smoke, fire, tides, etc.

Image analysis module 4102 may store obtained information in anyappropriate memory, such as memory unit 550 of apparatus 110 and/or 120,or remotely from apparatus 110 (e.g., stored in a server accessible toapparatus 110 via, for example, wireless transceiver 530). Any form ofdata structure, file system, or algorithm for storing the informationmay be employed.

Task prioritization module 4104 may store instructions executable byprocessor 210 and may handle various operations relating to creating atask, or changing or assigning task priority. For example, taskprioritization module 4104 may receive, from image analysis module 4102,information related to at least one of a plurality of tasks. Taskprioritization module 4104 may prioritize the tasks based on thereceived information. For example, task prioritization module 4104 mayassign a higher priority to a first task over the second task based atleast in part on information obtained by image analysis module 4102.

For example, tasks may be ranked by one or more factors such asimportance, urgency, duration, etc. Task prioritization module 4104 mayprioritize tasks based on a role or relationship between a user andanother person or object based on the obtained information. For example,task prioritization module 4104 may prioritize tasks based on adetermination that the user is a service provider and another person isa customer or vice versa.

Task prioritization module 4104 may assign or modify a score (such as anumeric score) associated with one or more of the plurality of tasks,based on the obtained information. For example, task prioritizationmodule 4104 may determine at least one score associated with acompleteness of one or more of the plurality of tasks and apply thescore to relevant tasks. A task that is nearly complete may beprioritized over a task that is not as nearly complete, or vice versa.Assigning a higher priority to a first task over a second task may bebased, at least in part, on maximizing an objective function. Forexample, prioritization module 4104 may assign priority to maximize anumber of tasks that may be completed in a particular amount of time orat a particular location. In some embodiments prioritization module 4104may assign priority to minimize or maximize a cost, risk, and/or otherfactor associated with completing a task.

Feedback module 4106 may store instructions executable by processor 210and may handle various operations relating to providing feedback to theuser based on the assigned priority. Feedback module 4106 may providefeedback in any appropriate way perceptible to a user, for example viatext, graphic display, audio, tactile feedback, etc.

In some embodiments, feedback module 4106 may provide feedback via adisplay associated with apparatus 110. In some embodiments, feedbackmodule 4106 may cause display of text, graphics, or other information ona display integrated in to apparatus 110. Any display technology knownin the art and suitable for inclusion in apparatus 110 may be used. Insome embodiments, feedback module 4106 may cause display of informationon one or more of a liquid crystal display (LCD), light-emitting diode(LED) display, organic light-emitting diode (OLED) display, etc.Additionally or alternatively, feedback module 4106 may display on adevice (e.g., computing device 120) paired with, networked with, orotherwise in communication with the wearable apparatus, such as a mobilecommunications device.

FIG. 42 is an example of a flowchart 4200 representing an exampleprocess for prioritizing tasks.

At step 4202, at least one processing device (e.g., processor 210) of awearable apparatus may analyze a plurality of images to obtaininformation related to at least some of the plurality of tasks. Asdiscussed earlier, the wearable apparatus may include a memory unit(e.g., memory 550) configured to store a plurality of tasks. In someembodiments, the plurality of tasks may comprise at least a first taskand a second task. The memory unit may store various informationregarding the tasks, including an indicator or description of the task,and may further include any other information related to or relevant tothe task (e.g., names, locations, due dates, goals, etc.).

FIG. 43A depicts an example of an image 4300 that may be analyzed by theat least one processing device at step 4202. In some embodiments, theimages may be analyzed by image analysis module 4102, as describedabove. For example, image 4300 depicts a pharmacy 4302. The at least oneprocessing device may analyze the image at step 4202 to identify, forexample, the text of “pharmacy” sign 4304 or pharmacy symbol 4306 (e.g.,information related to a location associated with one or more of aplurality of tasks). The at least one processing device may alsoidentify information such as the presence or number of vehicles 4308and/or parking spaces 4310 (e.g., information related to an estimatedtime to complete one or more of the plurality of tasks).

Information obtained at step 4202 may comprise information related to aperson associated with one or more of the plurality of tasks. Forexample, FIG. 43B depicts another example of an image 4340 that may beanalyzed by the at least one processing device at step 4202. Image 4340depicts two people, 4341 and 4342. The at least one processing devicemay identify person 4341 and/or person 4342 based on any trait or otherinformation that may be determined based on an analysis of image 4340,such as by facial analysis, OCR of a name tag 4350 or other textinformation such as the text 4348 displayed on person 4341′s apparel,etc.

The at least one processing device may determine relationshipinformation between people or objects at step 4202. For example, the atleast one processing device may determine based on any of the traitsshown in image 4340 that the user is a service provider and that person3431 and/or person 3432 is a customer or vice versa.

In some embodiments, the at least one processing device may determine aphysical, mental or emotional state of a person in an image. Forexample, the at least one processing device may identify traits such aswounds, the relative position of eyebrows 4344 and 4346 as indicatingthat the estimated patience level of person 4341 is higher than that ofperson 4342 or that the estimated stress level of person 4342 is higherthan that of person 4341.

Referring back to FIG. 42, at step 4204, the at least one processingdevice may prioritize tasks based on the obtained information. Forexample, at step 4204 the at least one processing device may assign ahigher priority to a first task over a second task. The prioritizationmay be performed by prioritization module 4104, as described above.

For example, referring back to FIG. 43A, based on a determination atstep 4202 that image 4300 includes pharmacy 4302, the at least oneprocessing device may assign a higher priority to a task associated witha pharmacy, such as picking up a prescription, than a task unrelated toa pharmacy.

The at least one processing device may take into account completeness ofa task at step 4204. For example, upon the presence of informationindicating that image 4300 includes pharmacy 4302, the at least oneprocessing device may prioritize a task of picking up a prescriptionbased on an amount of time that has elapsed since the prescription wasdropped off at the pharmacy.

At step 4204, the at least one processing deice may assign prioritybased, at least in part, on maximizing an objective function. Forexample, based on information related to a number of vehicles 4308and/or vacant parking spaces 4310, the at least one processing devicemay determine an estimated amount of time expected to be spent carryingout a task at pharmacy 4302. The at least one processing device may, forexample, prioritize a task related to pharmacy 4302 based on thisestimated amount of time (e.g., to minimize time spent carrying out thetask). However, if a user is late for a meeting, the priority of pickingup a prescription from the pharmacy may be reduced.

Referring back to FIG. 43B, the at least one processing device mayassign priority based on relationship information between people orobjects determined at step 4202. For example, the at least oneprocessing device may prioritize tasks based on a physical, mental oremotional state of a person in an image. For example, person 4341 andperson 4342 may both be customers and the user a service provider. Basedon the estimated patience level of person 4341 being higher than 4342and/or the estimated stress level of person 4342 being higher than thatof person 4341, the at least one processing device may prioritize a taskof providing service to person 4342 over a task of providing service toperson 4341.

At step 4206, the at least one processing device may provide feedback tothe user based on assigned priority. Providing the feedback may beperformed by feedback module 4106, as described above.

FIG. 43C depicts example interfaces 4360 and 4361. In an embodiment,interfaces 4360 and/or 4361 may be generated by feedback module 4106.Interface 4360 may be generated based on an increase in priority of atask to refill a prescription over one or more other tasks, for examplein response to analysis of and prioritization based on image 4300 ofFIG. 43A. Interface 4360 may include a message 4362 identifying thetask. In some embodiments, interface 4360 may include interface elements4364 and 4366. As shown in FIG. 43C, interface elements such as 4364 and4366 may provide options allowing a user to manipulate task statusand/or priority. In the example of FIG. 43C, interface element 4364 mayprovide functionality to mark the task complete. Interface element 4366may provide “remind me later” functionality. For example, selection ofinterface element 4366 may reduce the priority of the task, in effectassigning the task a lower priority than another task.

Interface 4361 may be generated based on a change in priority of a taskbased on image 4340 of FIG. 43B. For example, based on an assignment ofpriority at step 4204 of a task providing service to person 4342 over atask providing service to person 4341, interface 4361 may display amessage 4368 indicating the assignment of priority. Interface 4360 mayalso include an interface element 4370. As shown in FIG. 43C, interfaceelement 4370 may provide an option to confirm message 4368.

Wearable Apparatus for Analyzing Group Dynamics

In some embodiments, apparatus 110 may capture and analyze images todetermine group dynamics of detected persons in the environment of user100 of apparatus 110. For example, apparatus 110 may analyze images todetect two or more persons and analyze the images to determineassociation information related to the detected persons. Associationinformation may comprise information related to a relationship betweentwo or more persons based on an identity, a characteristic, or a socialsystem of an individual. The characteristic may relate to intrinsiccharacteristics such as age, gender, marital status, employment status,or extrinsic characteristics related to the surroundings of the persons,activities that that persons may be engaged in, emotional states of anindividuals in an environment, or relationships with third parties.Information for a first detected individual may be compared toinformation for a second detected individual to determine associationinformation. The association information may be analyzed to develop anunderstanding of a group dynamic. Data representing the associationinformation may be transmitted to a computing device (e.g., asmartphone, tablet, watch, computer, etc.) and/or a server for analysisand/or reporting to the user. The association information may also beused to update a social representation. The social representation maytake the form of a graphical representation showing changes inassociation information over time. The social representation update mayalso include displaying a captured image on a social network associatedwith user 100 through a smartphone application for user 100.

FIG. 44 illustrates an exemplary embodiment of a memory 4400 containingsoftware modules consistent with the present disclosure. Included inmemory 4400 are a monitoring module 4410, image processing module 4420,association module 4430, and database 4440. Modules 4410, 4420, and 4430may contain software instructions for execution by at least oneprocessing device, e.g., processor 210, included with apparatus 110. Insome embodiments, memory 4400 may comprise a non-transitory computerreadable medium that resides on a mobile communications device (e.g., asmart phone) configured to be wirelessly paired with wearable imagesensor 220 of capturing unit 710.

Monitoring module 4410 may provide for continuous, periodic, or ondemand monitoring. For example, monitoring may include tracking amovement of at least one detected person included in one or more imagescaptured by image sensor 220. For example, in one embodiment, apparatus110 may track a person as long as the object remains substantiallywithin the field-of-view of image sensor 220. In additional embodiments,monitoring module 4410 may continually orient image sensor 220 towards adetected person. For example, in one embodiment, monitoring module 4410may cause image sensor 220 to adjust an orientation to ensure that adetected person remains within the field-of view of image sensor 220. Inanother embodiment, monitoring module 4410 may continuously monitor anarea of interest included in one or more images captured by the imagesensor. For example, a user may be occupied by a certain task, forexample, delivering a speech to a large room, while image sensor 220remains oriented in a particular direction of a predetermined person ofinterest, such as a guest of honor or host of an event, and continuouslymonitors a portion of each image from a series of images to detect atrigger or other event. For example, image sensor 220 may be orientedtowards the person of interest and monitoring module 4410 may beconfigured to monitor the person's reaction to the speech while theuser's attention is directed to a piece of paper on placed on a lecternor a different area of the room. Monitoring module 4410 may operate intandem or concurrently with image processing module 4420 and associationmodule 4430 of FIG. 44.

Image processing module 4420 may be configured to identify at least twodetected persons from one or more images. In some embodiments, the atleast two detected persons may be identified through one or more imageprocessing techniques. For example, at least one image processingtechnique may be used to identify at least one feature of an image. Insome embodiments, an image processing technique may include one or moreof image enhancement, edge detection, image analysis, and dataextraction. Specific examples of the methods for identifying at leastone feature are exemplary only, and a person of ordinary skill in theart will recognize other methods for identifying the at least onefeature that remains consistent with the present disclosure. In someexamples, at least one feature may be detected using an object detectionalgorithm, using a neural network trained to detect objects and/orobjects in images, using a machine learning system, using a patternrecognition system, and so forth.

In some examples, at least one person may be identified in thesurroundings of the user using a facial recognition algorithm, using aneural network trained to detect faces and/or persons in images, and soforth. For example, a detected person may be identified by analyzing oneor more images for a characteristic of the detected person. Thecharacteristic of the individual may be any physical characteristic ofthe detected person including eye color, hair color, face shape,tattoos, piercings or others. The characteristics may be used toidentify the detected person as a person that user 100 has previouslyencountered and relate the present interaction with interactions storedin database 4440. Image processing module 4420 may also detect that thedetected person is unidentifiable based on no previous encounter and maygenerate a profile for the detected person, to be populated withinformation collected by image processing module 4420. Image processingmodule 4420 may additionally identify other objects in the surroundingsof user 100, for example, a book, a name tag, indicia of an activityetc. and populate the profile with data for the detected person withinformation to be used to determine group dynamics.

Association module 4430 may be configured to identify associationinformation based on features of two or more detected persons identifiedby image processing module 4410. Parameters for analyzing associationinformation may be configured and/or defined by user 100 and stored indatabase 4440. For example, user 100 may configure association module4430 to determine a category of information via a mobile application ona mobile communications device. For example, user 100 may plan to attenda professional event and configure image processing module 4420 toidentify and store employment information for each detected person basedon information identified from a name tag. Association module 4430 maybe configured to compare information for two or more detected persons todetermine association information. For example, image processing module4420 may communicate with database 4440 to identify a detected personbased on an online profile or social network of the detected person.Association module 4430 may then query the profile to acquire datarelated to each of the two detected persons. Association module 4430 maycompare the acquired data to determine association information for thetwo detected persons. For example, association module 4430 may determinethat the two detected persons are in the same professional industry,work for the same organization, or the like.

Monitoring module 4410, image processing module 4420 and associationmodule 4430 may be configured to transfer or receive data to/fromdatabase 4440 via network 240. In the disclosed embodiments, the databeing received from database 4440 may include numerous different typesof information based on the previously analyzed image data, storedrepresentational data, information related to a person, an identifiedlandmark, and any other information capable of being stored in oraccessed by database 4440. In some embodiments, data may be received andtransferred via computing device 120. Database 4440 and/or computingdevice 120 may retrieve information from different data sources (e.g., auser specific database, a user's social network account or otheraccount, the Internet, and other managed or accessible databases) andprovide information to apparatus 110 related to the analyzed image dataaccording to the disclosed embodiments. In some embodiments,calendar-related information retrieved from the different data sourcesmay be analyzed to provide certain time information or a time-basedcontext for providing certain information based on the analyzed imagedata.

FIGS. 45A-45D illustrate exemplary captured images for determiningassociation information. For example, in FIG. 45A, associationinformation may be based on a physical distance between two or morepersons in a group. Apparatus 110 may be configured to measure adistance between individuals. Continuous monitoring of distances betweenindividuals may also be used to detect changes in body language. Forexample, processor 210 may capture a plurality of images of two detectedpersons. Analysis of the plurality of images may result in a measuredaverage distance between two of more detected individuals. Distance 4510a may be of a smaller magnitude than distance 4510 b. Processor 210 maydetermine that persons separated by the smaller distance 4510 a areassociated, while persons separated by a distance 4510 b, which may belarger than a predetermined threshold, are not associated. Associationinformation based on physical distance may also include a determinationthat the two or more persons have had a previous encounter, arecolleagues, or have a familial bond. Association information may alsoprovide information about group dynamics including information about howfrequently a person is in close proximity to other people or howsociable a detected person is. For example, apparatus 110 may determinethat person 4520 is frequently within a close physical distance ofperson 4530. Apparatus 110 may determine that distance 4510 b betweenperson 4530 and a third-party changes at approximately the same rate andapproximately the same relative time as the distance between person 4520and the third-party. Based on this association information, processor210 may determine that person 4520 and person 4530 are attending anevent together (as professional partners, dates, etc.)

Association information based on a physical distance for analyzing groupdynamics may also be based on changes in physical distance between twodetected persons. Apparatus 110 may determine distance 4510 a anddistance 4510 c between person 4520 and 4530. Changes in distance 4510 cmay correlate with a body language indicator stored in a database. Forexample, apparatus 110 may detect that distance 4510 a is constant,while distance 4510 c increases. This may correlate with a stored bodylanguage indicating that person 4520 is leaning away from person 4530.Leaning away may suggest to processor 210 that person 4520 isexperiencing situational discomfort. Alternatively, a decrease indistance 4510 c followed by an increase may relate to an embrace betweenperson 4520 and 4530.

As illustrated in FIG. 45B, association information may also be based ona joint activity the at least two detected persons are engaged in.Apparatus 110 may detect an activity based on indicia of the activitythat is involved. The plurality of images captured by apparatus 110 maybe analyzed by comparing detected indicia of the activity with exemplaryindicia stored in a database. For example, apparatus 110 may capture aplurality of images representing an activity that two or more personsare engaged in, such as a sport, travel, attending a movie, etc. FIG.45B illustrates a plurality of persons engaged in a soccer game, asobserved by user 100 wearing apparatus 110. The plurality of images maydetermine that indicia 4540 is a soccer ball based on characteristics ofthe ball including shape, size, pattern, and color. Based on thisinformation, processor 210 may determine that the two or more detectedpersons are engaged in a soccer game. Apparatus 110 may also determineassociation information based on a team that a detected person isplaying for and identify the detected person's teammates. For example,apparatus 110 may determine that two or more detected persons play forthe same team based on a determination that the ball is frequentlypassed between the two or more detected persons. The determination mayalso be based on a determination that a first group of personsfrequently kick the ball in a first direction, while a second group ofpersons frequently kick the ball in the second direction, opposite tothe first direction.

As illustrated in FIG. 45C, association information may be based on ashared characteristic of the two detected persons. The sharedcharacteristic may include a detected intrinsic characteristic such ashair color, facial structure, height, etc. The shared characteristic mayprovide information relating to group dynamics. For example, apparatus110 may capture a plurality of images. Processor 210 may analyze thecaptured images and detect that the two or more persons in the capturedimage share a characteristic such as hair color or a unique nose shape.From this information, processor 210 may determine that detected personsare relatives. Processor 210 may detect that two of more detectedpersons share an intrinsic characteristic based on an analysis of thecaptured image.

The shared characteristic may also include situational characteristics.A detected situational characteristic may relate to a characteristicassociated with a given environment. For example, apparatus 110 maycapture a plurality of images. Processor 210 may analyze the capturedimages and detect that the two or more persons in the captured imageshare a situation characteristic such as a common article of clothing, asimilar accessory, or similar behaviors (e.g., laughing at the sametime, becoming agitated in response to a stimulus, etc.).

Shared characteristics may be used to analyze group dynamics. Forexample, processor 210 may determine that when a particular sharedcharacteristic is detected, a particular group dynamic is applied. Thisdetermination may be based on a set of rules for analyzing groupdynamics. For example, apparatus 110 may capture a plurality of imagesof a political rally. Processor 210 may detect two or more individualswith a shared characteristic, such as a similar pin or shirt worn bypersons with a political affiliation at the rally. The sharedcharacteristic may also include a facial reaction to a rallying cry.Processor 110 may also detect shared characteristics of a second groupdynamic. Two or more detected persons may become agitated in response toa rallying cry. This may indicate to processor 210 that members sharingthat characteristic may be protestors of the rally.

As illustrated in FIG. 45C, apparatus 110 of user 100 may capture aplurality of images of two or more persons. Upon detecting persons 4560a, 4560 b, and 4560 c, processor 210 may analyze the images to identifyshared characteristics. Processor 210 may determine similarities betweenfacial features 4580 a, 4580 b, and 4580 c. Similar facial features mayinclude a distinctive nose shape or eye color. Processor 210 may alsodetermine that persons 4560 a, 4560 b, and 4560 c all share a similarhair color. Other shared characteristics may include an identicalshopping bag 4570 a, 4570 b, and 4570 c, held by each detected person4560 a, 4560 b, and 4560 c. Based on these shared characteristics,processor 210 may determine that detected persons 4560 a, 4560 b, and4560 c are family members shopping together. Processor 210 may alsodetect other shared characteristics including departure from a store atthe same time, walking in a similar direction and remaining within apredetermined distance of each other. The shared characteristics mayprovide information about a group dynamic such as family members engagedin shopping.

Association information may also be based on a reaction of one of thetwo persons to an action of the other of the two persons. For example,user 100 of apparatus 110 may be engaged in a conversation or activitywith at least one other person. Apparatus 210 may capture a plurality ofimages representing different reactions of the detected persons to anaction of user 100. A reaction may include a departure of a detectedperson from the group, a facial reaction depicting an emotion, or bodylanguage indicators such as leaning in for an embrace.

As illustrated in FIG. 45D, user 100 of apparatus 110 may be engaged ina conversation with a group. Apparatus 110 may capture a plurality ofimages 4590 a, 4590 b, and 4590 c illustrating different reactions ofthe members of the group to an action by user 100. Image 4590 a showsthe detected person having a positive reaction to an action of user 100.Processor 210 may detect the positive reaction based on an analysis ofthe image showing that all detected persons are smiling Alternatively,an analysis of image 4590 b may show that some of detected persons havea positive reaction to user 100. Still alternatively, an analysis ofimage 4590 c may show that all of the detected persons have a negativereaction to an action by user 100. This analysis may be used tounderstand a group dynamic, particularly in response to another person.The group dynamic may indicate how well-liked a person is, howresponsive a group of persons are to a concept communicated to thegroup, or a general mood of the group.

FIG. 46 illustrates updating a social representation 4600 based ondetermined association information. Image 4590 a captured by apparatus110 may be analyzed to determine association information including apositive reaction to an action by user 100. In one embodiment, update tosocial representation 4600 includes a status update to a social networkassociated with user 100. Caption 4610 may be suggested based on thegroup dynamic detected by processor 210. Image 4590 a may be analyzed todetermine that the detected persons are smiling and thus have a positivereaction to user 100. Thus, the updated social representation may embodythe determined association information to illustrate a group dynamic.Updated social representation may also include expanding an onlinesocial network of user 100. For example, data representing theassociation information may be transmitted to an external computingdevice for analysis. The analysis may result in identifying an onlinesocial network account (e.g., Facebook, Twitter, LinkedIn, etc.) of thedetected person and transmit a request to user 110 to connect with thedetected person.

FIG. 47 illustrates a flowchart showing an exemplary process 4700 foranalyzing group dynamics related to user 100, consistent with disclosedembodiments. Apparatus 110 may implement one or more steps of process4700 to determine group dynamic information, illustrated in, for exampleFIGS. 45A-45D.

As illustrated in FIG. 47, at step 4710, apparatus 110 may capture aplurality of images from an environment of user 100. For example,capturing unit 710 may match the field-of-view of user 100 and imagesensors 220 of capturing unit 710 may capture images of one or moredetected persons in the field-of-view of user 100. Image sensors 220 mayalso capture images of the detected persons even when the attention ofuser 100 is directed away from the detected person. For example,monitoring module 4410 may continuously monitor one of the detectedpersons relative to the environment of the detected persons and user100. The environment of user 100 may include a room that user 100 hasentered or a path that user 100 is traveling. The environment may alsoinclude a maximum field-of-view of image sensor 220 in a direction thatit is pointed.

At step 4720, a processing device (e.g., processor 210) may perform afirst analysis of the plurality of images to detect at least two personsfrom the environment of the user. A person may be detected by apparatus110 using any one of various image processing techniques includingobject recognition, pattern recognition, or edge detection. Once aperson is detected in the environment, the person may be identified bycomparing the captured image with images stored in database 4440. Basedon the comparison, processor 210 may identify the person using imageprocessing module 4420 based on a previous stored encounter of theidentified person with user 100. For example, user 100 may be in a roomwith a plurality of persons. Apparatus 110 may capture a plurality ofimages of the room and using an image processing technique, detect oneor more persons in the room. Processor 210 may then use facialrecognition techniques to determine that user 100 has previouslyencountered the detected person based on a facial match to a storedimage of the detected person.

At step 4730, processor 210 may perform a second analysis of theplurality of images to determine association information related to theat least two detected persons. Association information may be based oncharacteristic information about the two or more detected personsidentified through facial recognition or biographical data stored on adatabase. The characteristic information may also be based oninformation about the environment of the detected persons. Further,image processing techniques may be used to detect environmentinformation. For example object identification may be used to identifyobjects within a predetermined proximity of detected persons. An objectin a portion of an image may be captured and analyzed by comparing thedetected object to stored images of representative objects in database4440. The characteristic information for a first detected individual maybe interrelated with characteristic information for a second detectedindividual to detect similarities or differences between the first andsecond detected persons. The similarities and differences betweencharacteristic information for the two or more detected persons may beused to determine association information. Association information mayalso be based on information obtained from a social network, or one ormore directories, such as a listing on a webpage. For example, processor210 may query social networks of the first and second detected personand determine that both are listed as “attending” an event hosted by athird detected person. Other information obtained from the socialnetwork of the detected persons may include a relationship status,employment status, a name of an employer, political affiliations,listing of other persons as family members, and biographical data. Thus,information about detected persons may be based on an online presence ofthe detected persons and may be used to determine associationinformation.

At step 4740, processor 210 may update a social representation based onthe determined association information. The social representation mayinclude providing an update to a user's social network account forlifelogging an event or updating stored data about the detected person.Social representations may also include information stored in database4440 regarding the detected persons. The information may includeprofiles for one or more detected persons and may be updated to includeassociation information from an encounter with user 100. For example,user 100 may attend a social event. Apparatus 110 may detect that two ormore persons maintained a physical distance from each other greater thana predetermined threshold value. Processor 210 may determine thatpreviously stored association data includes information that thedetected persons were regularly engaged in activity together, such asattending a movie and dining together. Processor 210 may update a socialrepresentation for each of the two detected person with new associationinformation indicating that the relationship between the two persons maybe strained. The updated social representation may also be a graphicalrepresentation of changes in association information over time. Theassociation information from a first event may be aggregated withassociation information from a second event to provide compositeassociation information for analysis.

Apparatus 110 may also transmit data to a computing device (e.g., asmartphone, tablet, watch, computer, etc.) over one or more networks viaany known wireless standard (e.g., cellular, Wi-Fi, Bluetooth®, etc.),or via a wired connection. The data transmitted by apparatus 110 mayinclude images, portions of images, identifiers related to informationappearing in analyzed images, association information, or group dynamicinformation. For example, an image may be analyzed and a group dynamicrelated to an activity occurring in the image may be identified. Theimage may then be transmitted to the computing device (e.g., a “paireddevice”). In the embodiments described herein, apparatus 110 may analyzeimages locally (on board the wearable apparatus) and/or remotely (via acomputing device). Further, a paired device may execute one or moreapplications (apps) to process, display, and/or analyze data (e.g.,identifiers, text, images, etc.) received from apparatus 110.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, orother optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1-196. (canceled)
 197. A wearable apparatus for capturing and processingimages, the wearable apparatus comprising: a wearable image sensorconfigured to capture a plurality of images from an environment of auser of the wearable apparatus; and at least one processing deviceprogrammed to: perform a first analysis of the plurality of images todetect at least two persons; perform a second analysis of the pluralityof images to determine association information related to the at leasttwo detected persons; and update a social representation based on thedetermined association information.
 198. The wearable apparatus of claim197, wherein the association information relates to a physical distancebetween the at least two persons.
 199. The wearable apparatus of claim197, wherein the association information is based on a joint activity ora shared characteristic the at least two persons are engaged in. 200.The wearable apparatus of claim 197, wherein the association informationincludes a characteristic based on information from a social network.201. The wearable apparatus of claim 200, wherein the sharedcharacteristic includes a distinguishing physical quality of the atleast two persons.
 202. The wearable apparatus of claim 200, wherein thedistinguishing physical quality includes a facial feature.
 203. Thewearable apparatus of claim 200, wherein the shared characteristicincludes a behavior response to a stimulus.
 204. The wearable apparatusof claim 197, wherein the association information includes a reaction ofone of the two persons to an action of the other of the two persons.205. The wearable apparatus of claim 204, wherein the reaction includesone of a positive, negative, or neutral reaction.
 206. The wearableapparatus of claim 197, wherein the second analysis further comprisescomparing a characteristic of a first detected person to acharacteristic of a second detected person.
 207. The wearable apparatusof claim 197, wherein the social representation includes a graphicalrepresentation showing changes in association information over time.208. The wearable apparatus of claim 197, wherein the socialrepresentation includes a social network of the user.
 209. A method forcapturing and processing images, the method comprising: obtaining, usinga wearable image sensor, a plurality of images captured from anenvironment of a user of a wearable apparatus; performing, using atleast one processor of the wearable apparatus, a first analysis of theplurality of images to detect at least two persons; performing, usingthe at least one processor, a second analysis of the plurality of imagesto determine association information related to the at least twodetected persons; and updating a social representation based on thedetermined association information.
 210. The method of claim 209,wherein the association information relates to a physical distancebetween the at least two persons.
 211. The method of claim 209, whereinthe association information is based on a joint activity or sharedcharacteristic the at least two persons are engaged in.
 212. The methodof claim 209, wherein the association information includes acharacteristic based on information from a social network.
 213. Themethod of claim 209, wherein the association information includes areaction of one of the two persons to an action of the other of the twopersons.
 214. A non-transitory computer readable medium containinginstructions that when executed by at least one processor cause the atleast one processor to perform steps, comprising: performing a firstanalysis of the plurality of images to detect at least two persons;performing a second analysis of the plurality of images to determineassociation information related to the at least two detected persons;and updating a social representation based on the determined associationinformation.
 215. The non-transitory computer-readable medium of claim214, wherein the association information includes a sharedcharacteristic of the two persons.
 216. The non-transitorycomputer-readable medium of claim 214, wherein the associationinformation includes a reaction of one of the two persons to an actionof the other of the two persons.